System and method for offering and billing advertisement opportunities

ABSTRACT

In various example embodiments, a system and method for offering and billing advertisement opportunities are provided. In example embodiments, an advertisement opportunity in media to be communicated to a user device associated with a viewer is identified. Attributes of the advertisement opportunity are determined. Based on these attributes, a value to assign to the advertisement opportunity is determined. The advertisement opportunity is offered to a potential advertiser based on the assigned value determined using the attributes of the advertisement opportunity.

RELATED APPLICATION

The present application is a continuation-in-part U.S. patentapplication Ser. No. 13/175,663, filed Jul. 1, 2011 and entitled“Advertising Methods for Advertising Time Slots and Embedded Objects,”which is a continuation of U.S. Pat. No. 7,979,877, issued Jul. 12, 2011and entitled “Advertising Methods for Advertising Time Slots andEmbedded Objects,” both of which applications are hereby incorporated byreference herein in their entirety.

FIELD

The present disclosure relates generally to advertising, and in aspecific example embodiment, to offering and billing advertisementopportunities.

BACKGROUND

Typically, an advertiser looks for advertising opportunities that may beof interest to the advertiser. For example, certain events may be moreappealing for the advertising of a product of the advertiser. However,the proactive searching of these opportunities by the advertiser may betime consuming.

BRIEF DESCRIPTION OF DRAWINGS

The appended drawings merely illustrate example embodiments of thepresent invention and cannot be considered as limiting its scope.

FIG. 1 is a schematic illustration of the overall system of the presentinvention;

FIG. 2 is a detailed block diagram of a set-top box, display, and remotecontrol device of the system of the present invention.

FIG. 3 is an illustration of an embodiment of the present inventionwherein four video signals may be combined into four composite videosignals.

FIG. 4 is an illustration of an embodiment of the present inventionwherein a main video image is combined with portions of a second videoimage to create five composite video signals.

FIG. 5 depicts another set top box embodiment of the present invention.

FIG. 6 depicts a sequence of steps employed to create a combined imageat a user's set top box.

FIG. 7 is an illustration of one implementation of the presentinvention.

FIG. 8 is an illustration of the manner in which interactive content maybe displayed.

FIG. 9 is a graphical representation of a database that provides a listof additional information associated with advertisers.

FIG. 10 is a graphical representation of a database that provides a listof attributes associated with advertisers.

FIGS. 11 and 12 are illustrations of scenes showing otherimplementations of the present invention.

FIGS. 13, 14, and 15 are illustrations of scenes showing otherimplementations of the present invention.

FIGS. 16 and 17 are illustrations of scenes showing otherimplementations of the present invention.

FIG. 18 is a schematic block diagram that generally illustrates oneembodiment of the present invention.

FIG. 19 is a schematic block diagram that generally illustrates anotherembodiment of the present invention.

FIG. 20 is a flow diagram that illustrates the steps that may beperformed in applying embedded ads into a video stream.

FIG. 21 is a flow diagram illustrating the steps that may be performedfor the installation of embedded ads by manual designation of availableadvertising space.

FIG. 22 is a flow diagram illustrating the steps that may be performedfor the installation of embedded ads by automatic designation ofavailable advertising space and searching for actual objects.

FIG. 23 is a flow diagram illustrating the steps that may be performedfor the installation of embedded ads by automatic designation ofavailable advertising space and searching for blank spaces.

FIG. 24 is a flow diagram illustrating the steps that may be performedfor the installation of embedded ads, using automatic methods, in amanner which combines the processes of looking for an actual object andblank space simultaneously.

FIG. 25 is a flow diagram illustrating the steps that may be performedfor the installation of embedded ads by using tags and markers todescribe advertising space.

FIG. 26 is a flow diagram illustrating a process of auctioning embeddedadvertising.

FIG. 27 is a flow diagram illustrating a process of biding on embeddedadvertising.

FIG. 28 is a flow diagram illustrating a process of modifying biddingstrategy based upon affinity data.

FIG. 29 is a flow diagram illustrating a process of bidding on liveevents

FIG. 30 is a flow diagram illustrating a process of analyzingadvertising opportunities for live events.

FIG. 31 is a flow diagram illustrating a process for billing advertisersbased upon display time of embedded ads.

FIG. 32 is a block diagram illustrating an example embodiment of anetwork architecture of a system used to manage offers and billing ofadvertisement opportunities.

FIG. 33 is a block diagram illustrating an example embodiment of acentral management system.

FIG. 34 is a block diagram illustrating an example embodiment of a salesengine.

FIG. 35 is a block diagram illustrating an example embodiment of abilling engine.

FIG. 36 is a flow diagram of an example high-level method for managingoffers and billing of advertisement.

FIG. 37 is a simplified block diagram of a machine in an example form ofa computing system within which a set of instructions for causing themachine to perform any one or more of the methodologies discussed hereinmay be executed.

DETAILED DESCRIPTION

The description that follows includes systems, methods, techniques,instruction sequences, and computing machine program products thatembody illustrative embodiments of the present invention. In thefollowing description, for purposes of explanation, numerous specificdetails are set forth in order to provide an understanding of variousembodiments. It will be evident, however, to those skilled in the artthat embodiments may be practiced without these specific details. Ingeneral, well-known instruction instances, protocols, structures, andtechniques have not been shown in detail.

Example embodiments described herein provide systems and methods foroffering and billing advertisement opportunities. In exampleembodiments, an advertisement opportunity in media to be communicated toa user device associated with a viewer is identified. Attributes of theadvertisement opportunity are determined. Based on these attributes, avalue to assign to the advertisement opportunity is determined. Theattributes may comprise a location of the advertisement opportunity in ascene of the media. The value assigned may be higher based on thislocation being a prominent location in the scene (e.g., an area where alevel of activity in the scene is higher than other areas of the scene,an area near a center of the scene). The attributes may also comprise aplot line attribute, whereby the value assigned to the advertisementopportunity is higher based on a scene being a critical scene in a plotline. Different weights may be applied to various attributes indetermining the value.

The advertisement opportunity is offered to a potential advertiser basedon the assigned value determined using the attributes of theadvertisement opportunity. The assigned value may be a starting bidprice in an auction for the advertisement opportunity, or the value maybe a fixed price in a fixed price offer for the advertisementopportunity. The potential advertisers may be identified by determiningpotential advertisers listing, in an advertiser database, attributesthat match the attributes of the advertisement opportunity.

Subsequently, the advertiser who wins the auction or responds to thefixed price offer will have their advertisement embedded into theadvertisement opportunity and is billed for the advertisement placement.The billing is based on the value assigned to the advertisementopportunity. In one example, a length of time an advertisement isdisplayed is determined and is factored into a calculation of a finalinvoice.

FIGS. 1 through 6 illustrate one method of generating embeddedadvertising using video combination techniques. The combination ofsignals may occur in the set top box or may occur in an upstream source(e.g., national broadcaster, local station). The embedded advertisingcan take the form of objects that are inserted into a video (e.g.,television) programming or other digital media using overlay pages thatare generated by markup language, by video combination techniques, or byany other desirable technique. Embedded advertising using overlay pagescan replace objects in, for example, the television broadcast bynegating the television broadcast display on a pixel by pixel basis sothat objects can actually be eliminated from the broadcast, such asdisclosed in U.S. patent application Ser. No. 10/212,289, entitled “PostProduction Visual Alterations,” by Thomas Lemmons, filed on Aug. 2,2002, which is specifically incorporated herein by reference for allthat it discloses and teaches. Alternatively, video combinationtechniques can be used to embed ads by combining video images at the settop box level, such as disclosed in U.S. patent application Ser. No.10/103,545, entitled “Video Combiner,” Thomas Lemmons, et al., filedMar. 20, 2002, which is specifically incorporated herein by referencefor all that it discloses and teaches. Inserted objects can track withthe background of the display and conform to objects in the display sothat the inserted objects appear as objects that are actually containedwithin the television broadcast such as disclosed in U.S. patentapplication Ser. No. 10/357,155, entitled “Tracking Moving Objects onVideo with Interactive Access Points”, filed Jan. 31, 2003, by Steven O.Markel which is specifically incorporated herein by reference for allthat it discloses and teaches. Embedded advertising provides a highdegree of flexibility for offering (e.g., auctioning or selling) ads.This high degree of flexibility allows television broadcasters to offerembedded advertising which can be bid upon by television advertisers forboth prerecorded and live programming.

FIG. 1 illustrates the interconnections of the various components thatmay be used to deliver a composite video signal to individual viewers.Video sources 100 and 126 send video signals 102 and 126 through adistribution network 104 to viewer's locations 111. Additionally,multiple interactive video servers 106 and 116 send video, HTML, andother attachments 108. The multiple feeds 110 are sent to several settop boxes 112, 118, and 122 connected to televisions 114, 120, and 124,respectively. The set top boxes 112 and 118 may be interactive set topboxes and set top box 122 may not have interactive features.

The video sources 100 and 126 and interactive video servers 106 and 116may be attached to a conventional cable television head-end, a satellitedistribution center, or other centralized distribution point for videosignals. The distribution network 104 may comprise a cable televisionnetwork, satellite television network, Internet video distributionnetwork, or any other network capable of distributing video ormultimedia data.

The interactive set top boxes 112 and 118 may communicate to theinteractive video servers 106 and 108 though the video distributionnetwork 104 if the video distribution network supports two-waycommunication, such as with cable modems. Additionally, communicationmay be through other upstream communication networks 130. Such upstreamnetworks may include a dial up modem, direct Internet connection, orother communication network that allows communication separate from thevideo distribution network 104.

Although FIG. 1 illustrates the use of interactive set-top boxes 112 and118, the present invention can be implemented without an interactiveconnection with an interactive video server, such as interactive videoservers 106 and 116. In that case, separate multiple video sources 100can provide multiple video feeds 110 to non-interactive set-top box 122at the viewer's locations 111. The difference between the interactiveset top boxes 112 and 118 and the non-interactive set top box 122 isthat the interactive set top boxes 112 and 118 incorporate thefunctionality to receive, format, and display interactive content andsend interactive requests to the interactive video servers 106 and 116.

The set top boxes 112, 118, and 122 may receive and decode two or morevideo feeds and combine the feeds to produce a composite video signalthat is displayed for the viewer. Such a composite video signal may bedifferent for each viewer, since the video signals may be combined inseveral different manners. The manner in which the signals are combinedis described in the presentation description. The presentationdescription may be provided through the interactive video servers 106and 116 or through another server 132. Server 132 may be a web server ora specialized data server.

As disclosed below, the set-top box includes multiple video decoders anda video controller that provides control signals for combining the videosignal that is displayed on the display 114. In accordance withcurrently available technology, the interactive set-top box 112 canprovide requests to the interactive video server 106 to provide variousweb connections for display on the display 114. Multiple interactivevideo servers 116 can provide multiple signals to the viewer's locations111.

The set top boxes 112, 118, and 122 may be a separate box thatphysically rests on top of a viewer's television set, may beincorporated into the television electronics, may be functions performedby a programmable computer, or may take on any other form. As such, aset top box refers to any receiving apparatus capable of receiving videosignals and employing a presentation description as disclosed herein.

The manner in which the video signals are to be combined is defined inthe presentation description. The presentation description may be aseparate file provided by the server 132, the interactive video servers106 and 116, or may be embedded into one or more of the multiple feeds110. A plurality of presentation descriptions may be transmitted andprogram code operating in a set top box may select one or more of thepresentation descriptions based upon an identifier in the presentationdescription(s). This allows presentation descriptions to be selectedthat correspond to set top box requirements and/or viewer preferences orother information. Further, demographic information may be employed byupstream equipment to determine a presentation description version for aspecific set top box or group of set top boxes and an identifier of thepresentation description version(s) may then be sent to the set top boxor boxes. Presentation descriptions may also be accessed across anetwork, such as the Internet, that may employ upstream communication ona cable system or other networks. In a similar manner, a set top box mayaccess a presentation description across a network that corresponds toset top box requirements and/or viewer preferences or other information.And in a similar manner as described above, demographic information maybe employed by upstream equipment to determine a presentationdescription version for a specific set top box or group of set top boxesand an identifier of the presentation description version(s) may then besent to the set top box or boxes. The identifier may comprise a URL,filename, extension or other information that identifies thepresentation description. Further, a plurality of presentationdescriptions may be transferred to a set top box and a viewer may selectversions of the presentation description. Alternatively, softwareprogram operating in the set top box may generate the presentationdescription and such generation may also employ viewer preferences ordemographic information.

In some cases, the presentation description may be provided by theviewer directly into the set top box 112, 118, 122, or may be modifiedby the viewer. Such a presentation description may be viewer preferencesstored in the set top box and created using menus, buttons on a remote,a graphical viewer interface, or any combination of the above. Othermethods of creating a local presentation description may also be used.

The presentation description may take the form of a markup languagewherein the format, look, and feel of a video image is controlled. Usingsuch a language, the manner in which two or more video images arecombined may be fully defined. The language may be similar to XML, HTMLor other graphical mark-up languages and allow certain video functionssuch as pixel by pixel replacement, rotation, translation, and deformingof portions of video images, the creation of text and other graphicalelements, overlaying and ghosting of one video image with another, colorkey replacement of one video image with another, and any other commandas may be contemplated. In contrast to hard-coded image placementchoices typical to picture-in-picture (PIP) display, the presentationdescription of the present invention is a “soft” description thatprovides freedom in the manner in which images are combined and that maybe easily created, changed, modified or updated. The presentation is notlimited to any specific format and may employ private or public formatsor a combination thereof. Further, the presentation description maycomprise a sequence of operations to be performed over a period of timeor over a number of frames. In other words, the presentation descriptionmay be dynamic. For example, a video image that is combined with anothervideo image may move across the screen, fade in or out, may be alteredin perspective from frame to frame, or may change in size.

Specific presentation descriptions may be created for each set top boxand tailored to each viewer. A general presentation description suitedto a plurality of set top boxes may be parsed, translated, interpreted,or otherwise altered to conform to the requirements of a specific settop box and/or to be tailored to correspond to a viewer demographic,preference, or other information. For example, advertisements may betargeted at selected groups of viewers or a viewer may have preferencesfor certain look and feel of a television program. In some instances,some presentation descriptions may be applied to large groups ofviewers.

The presentation descriptions may be transmitted from a server 132 toeach set top box through a backchannel 130 or other network connection,or may be embedded into one or more of the video signals sent to the settop box. Further, the presentation descriptions may be sent individuallyto each set top box based on the address of the specific set top box.Alternatively, a plurality of presentation descriptions may betransmitted and a set top box may select and store one of thepresentation descriptions based upon an identifier or other informationcontained in the presentation description. In some instances, the settop box may request a presentation description through the backchannel130 or through the video distribution network 104. At that point, aserver 132, interactive video server 106 or 116, or other source for apresentation description may send the requested presentation descriptionto the set top box.

Interactive content supplied by interactive video server 106 or 116 mayinclude the instructions for a set top box to request the presentationdescription from a server through a backchannel. A methodology fortransmitting and receiving this data is described in U.S. ProvisionalPatent Application entitled “Multicasting of Interactive Data Over aBack Channel”, filed Mar. 5, 2002 by Ian Zenoni, which is specificallyincorporated herein by reference for all it discloses and teaches.

The presentation description may contain the commands necessary forseveral combinations of video. In such a case, the local preferences ofthe viewer, stored in the set top box, may indicate which set ofcommands would be used to display the specific combination of videosuitable for that viewer. For example, in an advertisement campaign, apresentation description may include commands for combining severalvideo images for four different commercials for four different products.The viewer's preferences located inside the set top box may indicate apreference for the first commercial, thusly the commands required tocombine the video signals to produce the first commercial will beexecuted and the other three sets of commands will be ignored.

In operation, the device of FIG. 1 provides multiple video feeds 110 tothe viewer's locations 111. The multiple video feeds are combined byeach of the interactive set-top boxes 112, 118, 122 to generatecorrelated or composite video signals 115, 117, 119, respectively. Asdisclosed below, each of the interactive set-top boxes 112, 118, 122uses instructions provided by the video source 100, interactive videoservers 106, 116, a separate server 132, or viewer preferences stored atthe viewer's location to generate control signals to combine the signalsinto a correlated video signal. Additionally, presentation descriptioninformation provided by each of the interactive video servers 106, 116can provide layout descriptions for displaying a video attachment. Thecorrelated video signal may overlay the various video feeds on a fullscreen basis, or on portions of the screen display. In any event, thevarious video feeds may interrelate to each other in some fashion suchthat the displayed signal is a correlated video signal with interrelatedparts provided by each of the separate video feeds.

FIG. 2 is a detailed schematic block diagram of an interactive set-topbox together with a display 202 and remote control device 204. As shownin FIG. 2, a multiple video feed signal 206 is supplied to theinteractive set-top box 200. The multiple video feed signal 206 thatincludes a video signal, HTML signals, video attachments, a presentationdescription, and other information is applied to a tuner/decoder 208.The tuner/decoder 208 extracts each of the different signals such as avideo MPEG signal 210, an interactive video feed 212, another video orinteractive video feed 214, and the presentation description information216.

The presentation description information 216 is the informationnecessary for the video combiner 232 to combine the various portions ofmultiple video signals to form a composite video image. The presentationdescription information 216 can take many forms, such as an ATVEFtrigger or a markup language description using HTML or a similar format.Such information may be transmitted in a vertical blanking encodedsignal that includes instructions as to the manner in which to combinethe various video signals. For example, the presentation description maybe encoded in the vertical blanking interval (VBI) of stream 210. Thepresentation description may also include Internet addresses forconnecting to enhanced video web sites. The presentation descriptioninformation 216 may include specialized commands applicable tospecialized set top boxes, or may contain generic commands that areapplicable to a wide range of set top boxes. References made herein tothe ATVEF specification are made for illustrative purposes only, andsuch references should not be construed as an endorsement, in anymanner, of the ATVEF specification.

The presentation description information 216 may be a program that isembedded into one or more of the video signals in the multiple feed 206.In some cases, the presentation description information 216 may be sentto the set top box in a separate channel or communication format that isunrelated to the video signals being used to form the composite videoimage. For example, the presentation description information 216 maycome through a direct internet connection made through a cable modem, adial up internet access, a specialized data channel carried in themultiple feed 206, or any other communication method.

As also shown in FIG. 2, the video signal 210 is applied to a videodecoder 220 to decode the video signal and apply the digital videosignal to video RAM 222 for temporary storage. The video signal 210 maybe in the MPEG standard, wherein predictive and intracoded framescomprise the video signal. Other video standards may be used for thestorage and transmission of the video signal 210 while maintainingwithin the spirit and intent of the present invention. Similarly, videodecoder 224 receives the interactive video feed 212 that may comprise avideo attachment from an interactive web page. The video decoder 224decodes the video signal and applies it to a video RAM 226. Videodecoder 228 is connected to video RAM 230 and operates in the samefashion. The video decoders 220, 224, 228 may also perform decompressionfunctions to decompress MPEG or other compressed video signals. Each ofthe video signals from video RAMs 222, 226, 230 is applied to a videocombiner 232. Video combiner 232 may comprise a multiplexer or otherdevice for combining the video signals. The video combiner 232 operatesunder the control of control signals 234 that are generated by the videocontroller 218. In some embodiments of the present invention, ahigh-speed video decoder may process more than one video feed and thefunctions depicted for video decoders 220, 224, 228 and RAMs 222, 226,230 may be implemented in fewer components. Video combiner 232 mayinclude arithmetic and logical processing functions.

The video controller 218 receives the presentation descriptioninstructions 216 and generates the control signals 234 to control thevideo combiner 232. The control signals may include many commands tomerge one video image with another. Such commands may include directoverlay of one image with another, pixel by pixel replacement, colorkeyed replacement, the translation, rotation, or other movement of asection of video, ghosting of one image over another, or any othermanipulation of one image and combination with another as one mightdesire. For example, the presentation description instructions 216 mayindicate that the video signal 210 be displayed on full screen while theinteractive video feed 212 only be displayed on the top third portion ofthe screen.

The presentation description instructions 216 also instruct the videocontroller 218 as to how to display the pixel information. For example,the control signals 234 generated by the video controller 218 mayreplace the background video pixels of video 210 in the areas where theinteractive video feed 212 is applied on the top portion of the display.The presentation description instructions 216 may set limits as toreplacement of pixels based on color, intensity, or other factors.Pixels can also be displayed based upon the combined output of each ofthe video signals at any particular pixel location to provide a trulycombined output signal. Of course, any desired type of combination ofthe video signals can be obtained, as desired, to produce the combinedvideo signal 236 at the output of the video combiner 232. Also, anynumber of video signals can be combined by the video combiner 232 asillustrated in FIG. 2. It is only necessary that a presentationdescription 216 be provided so that the video controller 218 cangenerate the control signals 234 that instruct the video combiner 232 toproperly combine the various video signals.

The presentation description instructions 216 may include instructionssent from a server directly to the set top box 200 or the presentationdescription instructions 216 may be settable by the viewer. For example,if an advertisement were to be shown to a specific geographical area,such as to the viewers in a certain zip code, a set of presentationdescription instructions 216 may be embedded into the advertisementvideo instructing the set top box 200 to combine the video in a certainmanner or in a certain location.

In some embodiments, the viewer's preferences may be stored in the localpreferences 252 and used either alone or in conjunction with thepresentation description instructions 216. For example, the localpreferences may be to merge a certain preferred background with a newsshow. In another example, the viewer's local preferences may select froma list of several options presented in the presentation descriptioninformation 216. In such an example, the presentation descriptioninformation 216 may contain the instructions for several alternativepresentation schemes, one of which may be preferred by a viewer andcontained in the local preferences 252.

In some embodiments, the viewer's preferences may be stored in a centralserver. Such an embodiment may provide for the collection and analysisof statistics regarding viewer preferences. Further, customized andtargeted advertisements and programming preferences may be sent directlyto the viewer, based on their preferences analyzed on a central server.The server may have the capacity to download presentation descriptioninstructions 216 directly to the viewer's set top box. Such a downloadmay be pushed, wherein the server sends the presentation descriptioninstructions 216, or pulled, wherein the set top box requests thepresentation description instructions 216 from the server.

As also shown in FIG. 2, the combined video signal 236 is applied to aprimary rendering engine 238. The primary rendering engine 238 generatesthe correlated video signal 240. The primary rendering engine 238formats the digital combined video signal 236 to produce the correlatedvideo signal 240. If the display 202 is an analog display, the primaryrendering engine 238 also performs functions as a digital-to-analogconverter. If the display 202 is a high definition digital display, theprimary rendering engine 238 places the bits in the proper format in thecorrelated video signal 240 for display on the digital display.

FIG. 2 also discloses a remote control device 204 under the operation ofa viewer. The remote control device 204 operates in the standard fashionin which remote control devices interact with interactive set-top boxes,such as interactive set-top box 200. The set-top box includes a receiver242 such as an infrared (IR) receiver that receives the signal 241 fromthe remote 204. The receiver 242 transforms the IR signal into anelectrical signal that is applied to an encoder 244. The encoder 244encodes the signal into the proper format for transmission as aninteractive signal over the digital video distribution network 104. Thesignal is modulated by modulator 246 and up-converted by up-converter248 to the proper frequency. The up-converted signal is then applied toa directional coupler 250 for transmission on the multiple feed 206 tothe digital video distribution network 104. Other methods of interactingwith an interactive set top box may be also employed. For example,viewer input may come through a keyboard, mouse, joystick, or otherpointing or selecting device. Further, other forms of input, includingaudio and video may be used. The example of the remote control 204 isexemplary and not intended to limit the invention.

As also shown in FIG. 2, the tuner/decoder 208 may detect web addressinformation 215 that may be encoded in the video signal 102 (FIG. 1).This web address information may contain information as to one or moreweb sites that contain presentation descriptions that interrelates tothe video signal 102 and that can be used to provide the correlatedvideo signal 240. The decoder 208 detects the address information 215which may be encoded in any one of several different ways such as anATVEF trigger, as a tag in the vertical blanking interval (VBI), encodedin the back channel, embedded as a data PID (packet identifier) signalin a MPEG stream, or other encoding and transmitting method. Theinformation can also be encoded in streaming media in accordance withMicrosoft's ASF format. Encoding this information as an indicator ismore fully disclosed in U.S. patent application Ser. No. 10/076,950,filed Feb. 12, 2002 entitled “Video Tags and Markers,” which isspecifically incorporated herein by reference for all that it disclosesand teaches. The manner in which the tuner/decoder 208 can extract theone or more web addresses 215 is more fully disclosed in the abovereferenced patent application. In any event, the address information 215is applied to the encoder 244 and is encoded for transmission throughthe digital video distribution network 104 to an interactive videoserver. The signal is modulated by modulator 246 and up-converted byup-converter 248 for transmission to the directional coupler 250 overthe cable. In this fashion, video feeds can automatically be provided bythe video source 100 via the video signal 102.

The web address information that is provided can be selected, asreferenced above, by the viewer activating the remote control device204. The remote control device 204 can comprise a personalized remote,such as disclosed in U.S. patent Ser. No. 09/941,148, filed Aug. 27,2001 entitled “Personalized Remote Control,” which is specificallyincorporated by reference for all that it discloses and teaches.Additionally, interactivity using the remote 204 can be provided inaccordance with U.S. patent application Ser. No. 10/041,881, filed Oct.24, 2001 entitled “Creating On-Content Enhancements,” which isspecifically incorporated herein by reference for all that it disclosesand teaches. In other words, the remote 204 can be used to access “hotspots” on any one of the interactive video feeds to provide furtherinteractivity, such as the ability to order products and services, andother uses of the “hot spots” as disclosed in the above referencedpatent application. Preference data can also be provided in an automatedfashion based upon viewer preferences that have been learned by thesystem or are selected in a manual fashion using the remote controldevice in accordance with U.S. patent application Ser. No. 09/933,928,filed Aug. 21, 2001, entitled “iSelect Video” and U.S. patentapplication Ser. No. 10/080,996, filed Feb. 20, 2002 entitled “ContentBased Video Selection,” both of which are specifically incorporated byreference for all that they disclose and teach. In this fashion,automated or manually selected preferences can be provided to generatethe correlated video signal 240.

FIG. 3 illustrates an embodiment 300 of the present invention whereinfour video signals, 302, 304, 306, and 308, may be combined into fourcomposite video signals 310, 312, 314, and 316. The video signals 302and 304 represent advertisements for two different vehicles. Videosignal 302 shows an advertisement for a sedan model car, where videosignal 304 shows an advertisement for a minivan. The video signals 306and 308 are background images, where video signal 306 shows a backgroundfor a mountain scene and video signal 308 shows a background for anocean scene. The combination or composite of video signals 306 and 302yields signal 310, showing the sedan in front of a mountain scene.Similarly, the signals 312, 314, and 316 are composite video signals.

In the present embodiment, the selection of which composite image todisplay on a viewer's television may be made in part with a localpreference for the viewer and by the advertiser. For example, theadvertiser may wish to show a mountain scene to those viewers fortunateenough to live in the mountain states. The local preferences may dictatewhich car advertisement is selected. In the example, the localpreferences may determine that the viewer is an elderly couple with nochildren at home and thus may prefer to see an advertisement for a sedanrather than a minivan.

The methodology for combining the various video streams in the presentembodiment may be color key replacement. Color key replacement is amethod of selecting pixels that have a specific color and location andreplacing those pixels with the pixels of the same location from anothervideo image. Color key replacement is a common technique used in theindustry for merging two video images.

FIG. 4 illustrates an embodiment 400 of the present invention wherein amain video image 402 is combined with portions of a second video image404. The second video image 404 comprises four small video images 406,408, 410, and 412. The small images may be inserted into the main videoimage 402 to produce several composite video images 414, 416, 418, 420,and 422.

In the embodiment 400, the main video image 402 comprises a border 424and a center advertisement 426. In this case, the border describestoday's special for Tom's Market. The special is the centeradvertisement 426, which is shrimp. Other special items are shown in thesecond video image 404, such as fish 406, ham 408, soda 410, and steak412. The viewer preferences may dictate which composite video is shownto a specific viewer. For example, if the viewer is a vegetarian,neither the ham 408 nor steak 412 advertisements would be appropriate.If the person had a religious preference that indicated that they wouldeat fish on a specific day of the week, for example, the fish special406 may be offered. If the viewer's preferences indicated that theviewer had purchased soda from the advertised store in the past, thesoda advertisement 410 may be shown. In cases where no preference isshown, a random selection may be made by the set top box, a defaultadvertisement, or other method for selecting an advertisement may beused. While FIG. 4 provides the center advertisement 426, it is notedthat the advertisement may be located anywhere on the main video image402.

Hence, the present invention provides a system in which a correlated orcomposite video signal can be generated at the viewer location. Anadvantage of such a system is that multiple video feeds can be providedand combined as desired at the viewer's location. This eliminates theneed for generating separate combined video signals at a productionlevel and transmission of those separate combined video signals over atransmission link. For example, if ten separate video feeds are providedover the transmission link, a total of ten factorial combined signalscan be generated at the viewer's locations. This greatly reduces thenumber of signals that have to be transmitted over the transmissionlink.

Further, the present invention provides for interactivity in both anautomated, semi-automated, and manual manner by providing interactivevideo feeds to the viewer location. As such, greater flexibility can beprovided for generating a correlated video signal.

FIG. 5 depicts another set top box embodiment of the present invention.Set top box 500 comprises tuner/decoder 502, decoder 504, memory 506,processor 508, optional network interface 510, video output unit 512,and user interface 514. Tuner/decoder 502 receives a broadcast thatcomprises at least two video signals 501. In one embodiment of FIG. 5,the tuner/decoder 502 is capable of tuning at least two independentfrequencies. In another embodiment of FIG. 5, the tuner/decoder 502decodes at least two video signals contained within a broadcast band, asmay occur with QAM or QPSK transmission over analog television channelbands or satellite bands. “Tuning” of video signals may compriseidentifying packets with predetermined PID (Packet Identifiers) valuesor a range thereof and forwarding such packets to processor 508 or todecoder 504. For example, audio and/or video data packets 503 may betransferred to decoder 504 and control packets may be transferred toprocessor 508. Data packets may be discerned from control packets 507through secondary PIDs or through PID values in a predetermined range.Decoder 504 processes packets received from tuner/decoder 502 andgenerates and stores image and/or audio information 505 in memory 506.Image and audio information 505 may comprise various information typescommon to DCT based image compression methods, such as MPEG and motionJPEG, for example, or common to other compression methods such aswavelets and the like. Audio information may conform to MPEG or otherformats such as those developed by Dolby Laboratories and THX as arecommon to theaters and home entertainment systems. Decoder 504 maycomprise one or more decoder chips to provide sufficient processingcapability to process two or more video streams substantiallysimultaneously. Control packets 507 provided to processor 508 mayinclude presentation description information. Presentation descriptioninformation may also be accessed employing network interface 510.Network interface 510 may be connected to any type of network 511 thatprovides access to a presentation description including modems, cablemodems, DSL modems, upstream channels in a set top box and the like.Network interface 510 may also be employed to provide user responses tointeractive content to an associated server or other equipment.

Processor 508 of FIG. 5 employs the presentation description to controlthe combination of the image and/or audio information stored in memory506. The combining process may employ processor 508, decoder 504, or acombination of processor 508 and decoder 504. The combined image and oraudio information, as created employing the presentation description, issupplied to video output unit 512 that produces and output signal for atelevision, monitor, or other type of display 515. The output signal 516may comprise composite video, S-video, RGB, or any other format. Userinterface 514 supports a remote control, mouse, keyboard or other inputdevice. User input 517 may serve to select versions of a presentationdescription or to modify a presentation description via user interface514.

FIG. 6 depicts a sequence of steps 600 employed to create a combinedimage at a user's set top box. At step 602 a plurality of video signalsare received. These signals may contain digitally encoded image andaudio data. At step 604 a presentation description is accessed. Thepresentation description may be part of a broadcast signal or may beaccessed across a network. At step 606, at least two of the videosignals are decoded and image data and audio data (if present) for eachvideo signal is stored in a memory of the set top box. At step 608,portions of the video images and optionally portions of the audio dataare combined in accordance with the presentation description. Thecombination of video images and optionally audio data may producecombined data in the memory of the set top box, or such combination maybe performed “on the fly” wherein real-time combination is performed andthe output provided to step 610. For example, if a mask is employed toselect between portions of two images, non-sequential addressing of theset top box memory may be employed to access portions of each image in areal-time manner, eliminating the need to create a final display imagein set top box memory. At step 610 the combined image and optionallycombined audio are output to a presentation device such as a television,monitor, or other display device. Audio may be provided to thepresentation device or to an amplifier, stereo system, or other audioequipment.

The presentation description of the present invention provides adescription through which the method and manner in which images and/oraudio streams are combined may be easily be defined and controlled. Thepresentation description may specify the images to be combined, thescene locations at which images are combined, the type of operation oroperations to be performed to combine the images, and the start andduration of display of combined images. Further, the presentationdescription may include dynamic variables that control aspects ofdisplay such as movement, gradually changing perspective, and similartemporal or frame varying processes that provide image modification thatcorresponds to changes in scenes to which the image is applied.

Images to be combined may be processed prior to transmission or may beprocessed at a set top box prior to display or both. For example, animage that is combined with a scene as the scene is panned may beclipped to render the portion corresponding to the displayed image suchthat a single image may be employed for a plurality of video frames.

The combination of video images may comprise the process of replacingand/or combining a portion of a first video image with a second videoimage. The manner in which images are combined may employ any hardwareor software methods and may include bit-BLTs (bit block logictransfers), raster-ops, and any other logical or mathematical operationsincluding but not limited to maxima, minima, averages, gradients, andthe like. Such methods may also include determining an intensity orcolor of an area of a first image and applying the intensity or color toan area of a second image. A color or set of colors may be used tospecify which pixels of a first image are to be replaced by or to becombined with a portion of a second image. The presentation descriptionmay also comprise a mask that defines which areas of the first image areto be combined with or replaced by a second image. The mask may be asingle bit per pixel, as may be used to specify replacement, or maycomprise more than one bit per pixel wherein the plurality of bits foreach pixel may specify the manner in which the images are combined, suchas mix level or intensity, for example. The mask may be implemented aspart of a markup language page, such as HTML or XML, for example. Any ofthe processing methods disclosed herein may further include processesthat produce blurs to match focus or motion blur. Processing methods mayalso include processes to match “graininess” of a first image. Asmentioned above, images are not constrained in format type and are notlimited in methods of combination.

The combination of video signals may employ program code that is loadedinto a set top box and that serves to process or interpret apresentation description and that may provide processing routines usedto combine images and/or audio in a manner described by the presentationdescription. This program code may be termed image combination code andmay include executable code to support any of the aforementioned methodsof combination. Image combination code may be specific to each type ofset top box.

The combination of video signals may also comprise the combination ofassociated audio streams and may include mixing or replacement of audio.For example, an ocean background scene may include sounds such as birdsand surf crashing. As with video images, audio may be selected inresponse to viewer demographics or preferences. The presentationdescription may specify a mix level that varies in time or across aplurality of frames. Mixing of audio may also comprise processing audiosignals to provide multi-channel audio such as surround sound or otherencoded formats.

Embodiments of the present invention may be employed to add content toexisting video programs. The added content may take the form ofadditional description, humorous audio, text, or graphics, statistics,trivia, and the like. As previously disclosed, a video feed may be aninteractive feed such that the viewer may response to displayed imagesor sounds. Methods for rendering and receiving responses to interactiveelements may employ any methods and includes those disclosed inincorporated applications. Methods employed may also include thosedisclosed in U.S. application Ser. No. 10/403,317 filed Mar. 27, 2003 byThomas Lemmons entitled “Post Production Visual Enhancement Rendering”,which is a continuation-in-part of U.S. patent application Ser. No.10/212,289 filed Aug. 8, 2002 by Thomas Lemmons entitled “PostProduction Visual Alterations”, which is based upon U.S. ProvisionalPatent Application No. 60/309,714 filed Aug. 8, 2001 by Thomas Lemmonsentitled “Post Production Visual Alterations”, all of which arespecifically incorporated herein for all that they teach and disclose.As such, an interactive video feed that includes interactive contentcomprising a hotspot, button, or other interactive element, may becombined with another video feed and displayed, and a user response theinteractive area may be received and may be transferred over theInternet, upstream connection, or other network to an associated server.

FIGS. 7 through 25 illustrate, in more detail, the manner in whichenhancements, such as HTML overlay pages, can be used to create embeddedadvertising in video programming.

FIG. 7 is an illustration of one implementation for providing embeddedadvertising using enhancements. Referring to FIG. 7, a scene from avideo broadcast is displayed showing various items or objects in thescene. The video may be displayed on a display device 700, such as atelevision set, which may comprise any type of display such as a plasmadisplay, flat panel display, CRT monitor, or any other display. Thevideo may also be displayed on a computer monitor displaying embeddedads within video streams over the Internet, or the video may bedisplayed on any other type of display system/device including moviesdisplayed at movie theaters. In the scene, a person 702 is shown assitting on a chair 704 at a table 706. The person 702 is seated in frontof an object, such as a pizza box 708, from which the person is eatingpizza. The object, or pizza box, may be automatically identified by someexisting technique such as pattern recognition software, as will bediscussed in more detail in FIG. 23. Alternatively, tags and markers, asdisclosed in the above referenced patent applications, may be suppliedin the video stream that provides the object identification information(pizza box) and location of the object in the video stream. The systemmay then determine if a pizza advertiser has been listed as an embeddedadvertiser. That is, a list may be compiled that comprises advertiserswho wish to exhibit advertisements, in the form of embedded advertising,in a video presentation. The list of advertisers may also include theitem (attribute), which may be in the form of a product, idea, etc.,that the advertisers wish to promote. Such a list of advertisers andattributes may be stored in a database by the broadcaster. The list ofadvertisers and attributes is more fully disclosed with regard to thedescription of FIG. 9.

It is noted that due to the prominent location and activity involvingthe pizza box in the above example, a premium value or price may berequested for placement of an advertisement on the pizza box. Thepremium value may be in the form of a higher starting bid price or ahigher fixed price as compared to a price for an ad location locatedaway from the activity (e.g., a corner or bottom of the screen).

Referring again to FIG. 7, once an object, in this instance a pizza box708, is recognized, the advertiser list may be queried for a pizzaadvertiser. If a pizza advertiser has been listed as an embedded adadvertiser, an embedded ad, which may be in the form of a logo, text, orother form of advertisement for the advertiser, is graphically displayed(superimposed or overlaid) on the pizza box. The process of applying, orsuperimposing, the embedded on the pizza box is discussed in furtherdetail with regard to the description of FIG. 23. The embedded ad maycomprise any form of video enhancement, including video sequences, text,graphical images that may be in bitmap format or any other format, orother graphical representation. If the embedded ad is a “hot label,”(i.e., the embedded ad is an interactive enhancement), the viewer maythen click on the hot label to order a pizza from that advertised store.Details of this process are further discussed with respect to FIG. 23.

Embedded ads may comprise enhancements to the original video signal,such as improvements, augmentations or new additions. For example, acompany logo may be applied to a soda-can. The soda-can may alreadyexist within the original video signal, or the soda-can may be insertedinto the original signal. Alternatively, changes may comprise downgradesfrom the original video signal, such as degradations, reductions, ordeletions. For example, a portion of the video may be “erased” in orderto create a blank space, which may be used for applying embedded ads ordisplaying advertisements. Object insertion, in this fashion, can bedone using either video combination techniques, such as disclosed abovein FIGS. 1 through 7, or by using enhancements, such as markup languageoverlay pages, including interactive enhancements.

Embedded ads may be applied to any type of video content and videosignals such as live or taped a television broadcasts, animation, filmedvideo and the like or may be captured as a magnetic or optical mediafile, such as in a Digital Video/Versatile/Variable Disc (DVD)feature-length film. Embedded ads may also comprise informationalcontent such as markings. Content may be created by taping or filming,such as the filming of a live broadcast, a feature-length film, atelevision program, or other presentation. Content may also be createdusing animation. Cartoons on television or on feature-length films areone example of animation. Content creation may comprise computeranimation. Computer animation is a principal component of multimediapresentations including feature-length films, and many softwareapplications can be used to create animations that can be displayed onany display device, such as a computer monitor, television screen, ormovie screen.

Of course, embedded ads may be applied to any form of viewing material,including educational programs, multimedia presentations, orentertainment presentations. Such multimedia or entertainmentpresentations may include digitally enhanced or computer enhancedpresentations, animated presentations, which may include feature-lengthanimated films, animated programs that are broadcast over a televisionnetwork, DVD animated presentations, or other forms of animatedpresentations. Additionally, embedded ads may be applied to suchmultimedia or entertainment presentations as feature-length video filmsthat may be shown in movie theatres, perhaps in a digital format, overfiber-optic connections, through laser technology, or through othertechnology.

As mentioned above, techniques to facilitate supplementary content in avideo signal can also be accomplished by combining video signals. Such atechnique is disclosed in U.S. Provisional Application Ser. No.60/278,669, filed Mar. 20, 2001 entitled “DELIVERY OF INTERACTIVE VIDEOCONTENT USING FULL MOTION VIDEO PLANES”, now U.S. Non-Provisionalapplication Ser. No. 10/103,545, filed Mar. 20, 2002 entitled “VIDEOCOMBINER”, both of which are specifically incorporated herein byreference for all that they discloses and teach. In accordance with theabove referenced application, the system is capable of combining videosignals that incorporate embedded ads at the viewer's location. Forexample, multiple video feeds, some of which can include embedded ads,are provided to a viewer's set-top box together with instructions forcombining two or more video feeds. The video feeds can then be combinedin a set-top box or otherwise located at or near the viewer's locationto generate the combined or correlated video signal for display withembedded ads. As mentioned above, the video feeds can be combined by theTV broadcaster prior to transmission over the video distributionnetwork.

Additionally, one or more video feeds can comprise enhanced video thatis provided from an Internet connection. HTML-like scripting can be usedto indicate the layout of the enhanced video signal. Instructions can beprovided for replacement of particular segments comprising the embeddedads space, to make an interactive embedded advertisement.

Hot labels (interactive embedded ads) can appear unobtrusively during avideo broadcast. As disclosed above, the embedded ad itself can be aninteractive element which is placed on specific objects that relate tothe advertising element or in a blank portion of the screen. As alsodisclosed above, embedded ads may not be interactive (e.g., embedded adsmay exist simply as an advertisement or informational material that ispart of the video stream). Insertion of video content to producecombined video streams, as well as insertion of interactive content canbe accomplished by using tags and markers inserted into the video streamas disclosed in U.S. Non-Provisional application Ser. No. 10/076,950,filed Feb. 12, 2002, which is based upon U.S. Provisional ApplicationSer. No. 60/268,350, filed Feb. 12, 2001 entitled “VIDEO TAGS ANDMARKERS”, both of which are specifically incorporated herein byreference for all that they disclose and teach.

Embedded ads that are interactive links can allow for superimposition ofinteractive embedded ad information on products. Embedded ads may appearas a visual enhancement, such as a logo or name, and thus anadvertisement of any party or advertiser, such as a local or nationalcompany, or simply informational. Embedded ads can also beindividualized, localized advertisements. An embodiment may employpattern recognition techniques that provide an automated method ofidentifying objects on which to place embedded ads that pertain to theobject. Embedded ads may be localized to a point where specificadvertisements can be superimposed from set top-boxes, so thatadvertising can be personalized down to the household or individualviewing level.

Embedded ads may be applied to content by creating “embedded ad space,”or space within the content (which may be a video stream), that isavailable for application of an embedded ad. The designation of suchembedded ad space may be completed by manual or automatic methods. Theembedded ad or type of embedded ad to be superimposed may be chosenbased on “attributes,” as further explained with regard to thedescription of FIG. 9. A list of advertisers, along with theadvertisers' associated attributes, may be stored in a database.“Attributes” are a class of actual objects, or blank spaces that may beassociated with an advertiser or with types or classes of information.An embedded ad may be applied, to the embedded ad space within the videostream, according to the class of object or blank space identifiedwithin the video stream. The identified object or blank space (withinthe video stream) may be compared to objects, classes of objects orblank spaces listed as attributes within the aforementioned database.Once a matching actual object or blank space is found, an appropriateembedded ad is chosen and applied to the embedded ad space within thevideo stream.

Referring again to FIG. 7, the embedded ad 710 appears as a logo of alocal franchise called “Paul's Pizzeria.” The embedded ad may be anactual logo, name, or advertisement of a company that can be either alocal company or national chain and may exist as a non-obtrusiveadvertisement that appears to be an actual element of the originallyfilmed scene. The embedded ad 710 appears as a logo on the actual pizzabox, thus appearing completely naturally in the scene. The embedded ad710 may exist as a non-interactive enhancement that is part of the videosignal, and thus the embedded ad may simply exist as an advertisement,or the embedded ad may be an interactive element that has thefunctionality to be activated by the viewer, thus acting as anadvertisement and interactive tool.

Again referring to FIG. 7, if the embedded ad 710 is interactive, theembedded ad may be activated by the viewer 712. As previously described,the “viewer” is a person watching the video. Activation of the embeddedad 710 may occur in several ways, including toggling through a series ofembedded ads using control buttons on some form of a hand-held controldevice 716, such as a remote-control, activating a cursor 718 using awired or wireless mouse, or by other means including but not by way oflimitation, light beams, such as IR beams that can be used as pointersto point at embedded ads. Such a system may operate in a fashion similarto a touch screen that identifies the light beam rather than a user'sfinger. A cursor 718 may be an indicator, such as a spot, a flashingline, an arrow, a symbol, or any other indicator, that is displayed onthe TV screen 700. The cursor position may be controlled by another formof a hand-held control device, such as a wired mouse or a wireless mouse(not shown) in a similar fashion to that of a mouse being connected to acomputer and monitor. Once activated, the embedded ad may make availablea host of options, including bringing up an order form 720, which mayallow the viewer to begin the process of directly ordering a pizza. Theorder form 720 may appear on an unused portion of the TV screen 700,thereby causing no interruption of the current video presentation.

FIG. 8 is an illustration of the manner in which interactive content maybe displayed. Once activated, the embedded ad (as referred to in FIG. 7)may display information in an enhanced content window or border area801. The current video presentation shown on display device 800 mayremain viewable in the broadcast window 840. As shown in FIG. 8, thedisplayed video that includes the same objects that appeared in FIG. 7,including the table 826, pizza box 828, embedded ad 830, person 832, andchair 834, remain visible. Alternatively, the information brought up bythe embedded ad may be displayed on a completely separate page. Theembedded ad may bring up a graphical user interface (GUI) 802, a textbox, or some other presentation scheme that serves to presentinformation in some manner to the viewer. The GUI 802 may presentadditional information 804 about the company for which the embedded adhas been activated. Such additional information may be submitted by theadvertiser or by the author (e.g., person examining and/or editing thevideo signal). The additional information 804 may be contained in aseparate storage device, such as a database. The database containingadditional advertiser information is more fully disclosed with regard tothe description of FIG. 9. Once an embedded ad is activated, thiscorresponding database can be accessed for the additional information.The separate database may be located in a central computer, or onmultiple computers, which may be a server at the head-end, aretransmission site, other site, or in the household of the viewer in aset-top box. The central computer is more fully disclosed with regard tothe description of FIGS. 17 and 18. The central computer inserts theembedded ad. Insertion of the embedded ad is discussed in further detailin FIG. 23.

Referring again to FIG. 8, the additional information 804 may includecompany profile 806, web address 808, coupons 810, or other information.The web address 808 may simply be an address that is non-interactive andis simply there for the viewer to take note of and to have theopportunity to visit that address at a later time, or the web addressmay act as a direct, interactive link. The interactive link 808 may linkthe viewer to the company home page, or the company website that allowsthe viewer to make a purchase 814. That is, the “order a pizza” website814 may be a special website designed specifically for viewerinteraction with embedded ads. In addition, if the advertiser is anational company, the additional information may include localdistributors or franchises 812 of the national company. The embedded admay provide a link to the local server to provide information regardingthe local store. For example, if Paul's Pizzeria is a national chain,the local broadcaster may superimpose the local franchise of Paul'sPizzeria at the server at the head-end, as illustrated in FIG. 17. Theviewer, when clicking on the embedded ad, will see their local Paul's'Pizzeria in the additional information display and be able to order apizza directly from their local store.

Furthermore, the viewer may also have the choice of accessing links thathave nothing to do with the current video presentation. For example, ona different part of the graphical user interface 802, the viewer may beable to activate links 816 that facilitate access to the world-wide-web.For example, the viewer may be able to cast a vote 818 on whichcharacter the viewer liked the best during a particular televisionprogram such as the “Survivor” series. The viewer may also be able tocast a vote 820 for elections, read about their local politicalrepresentatives by clicking on an interactive link 822 and enteringtheir local information, or even read about current world events 824.

FIG. 9 is a graphical representation of a database that provides a listof additional information associated with advertisers, as listed indatabase 900. As previously mentioned with regard to the description ofFIG. 8, the additional information of an advertiser may be contained ina storage device, such as a database 900. As illustrated in FIG. 9, suchadditional information may be submitted as an entry 902. For example,one entry may be for the advertiser's name, Paul's Pizzeria 901, withadditional associated information that may include company profile, webaddress, coupons, or other information. The additional information maybe submitted by the advertiser and/or by the author. Additionally, otheradvertisers may be stored in the same database along with similarinformation, as a separate entry 904.

FIG. 10 is a graphical representation of a database that provides a listof attributes associated with advertisers, as listed in database 1006.Database 1006 may also contain an embedded ad itself, or a link to anembedded ad. Referring to FIG. 10, in order to paint the embedded adonto a particular object or even onto a blank space, there is adetermination of whether or not there is an appropriate embedded ad toapply. Thus, a list of advertisers is stored in a storage device, suchas a database 1006, along with associated attributes. “Attributes” aredefined as any actual object or categories of objects that areassociated with the advertiser, or “blank space,” which willsubsequently be explained. Attributes may be submitted by the advertiserand/or by the author. Referring to FIG. 10, an advertiser, as may beseen as one entry out of many, is stored in the database 1006. Forexample, the advertiser General Electric 1014 may be an entry 1008.General Electric may be accompanied by a list of attributes 1010. Suchattributes 1010 may include actual objects, such as “stove,”“refrigerator,” “toaster,” and other actual objects or categories suchas appliances, etc. Alternatively, advertisers may wish to have theopportunity to advertise on an available blank space; in this case, theadvertiser may submit “blank space” as one of their attributes. Forexample, the advertiser Joe's Appliances 1016, as seen in entry 1012,may elect to list “blank space” 1014 as an attribute, which would allowJoe's Appliances the opportunity to bid on a blank space in the videocontent as well as to apply an advertisement to an actual object. Blankspace may be further constrained to specific areas such as the dasherboards below the Colorado Avalanche hockey bench. Blank spaceadvertising is more fully disclosed with regard to the description ofFIGS. 11 and 12. Additionally, other advertisers may be stored in thesame database along with similar information, as other entries such as1018 and 1020. In addition, this database may store the file name ofand/or link to an embedded ad graphics file, which may be stored in aseparate location. Alternatively, the database may contain embedded adtext files 1026, 1028. These concepts are more fully disclosed withrespect to the description of FIGS. 13 and 14. The manner in which acompany is chosen as an advertiser using an embedded ad is discussed ingreater detail in FIG. 24. If the system of the present invention findsa pizza advertiser in the list of advertisers, an embedded ad is paintedonto the identified pizza box, at which time the viewer may click on theembedded ad and order a pizza. The process of applying the embedded adis discussed in greater detail in FIG. 23.

FIGS. 11 and 12 are illustrations of scenes showing other embodiments.Embedded ads can be coded to change on a per-scene basis. As illustratedby FIG. 11, in scene one 1100 of a television show, a refrigerator 1102may be superimposed (e.g., overlaid) with an embedded ad 1104. In scenetwo 1206, the camera angle is different, which changes the position ofthe refrigerator 1208. During this scene, the embedded ad 1210 remainson the refrigerator. Furthermore, instead of the embedded ad appearingas a normal part of the scene as illustrated by embedded ads 1104 and1210, an embedded ad 1212 may appear larger and more apparent, such astext written outside the boundaries of the area rug 1214, thus appearingto the viewer as an obvious part of the scene.

FIGS. 13, 14 and 15 are illustrations of scenes showing otherembodiments. As shown in FIG. 13, a football game is displayed on adisplay screen 1300. The scene illustrates a football field 1302 andviewing stands with real banner ads 1304, 1306 and 1308 as typicallyfound in sports arenas. Embedded advertising is supplemented in thispre-existing video signal by superimposing embedded ads 1310, 1312, and1314 on to the playing field 1302. Thus, the final enhanced video signalappears to the viewer as though the field embedded ads 1310, 1312, and1314 are inherent to the setting and look as if they are a natural partof the scene.

In example embodiments, different ad locations on the playing field maybe valued differently. For example, because the action in the footballgame is centered around a location of the ad 1310, that ad location maycost more for the advertiser (e.g., have a higher value) than an adlocation further away from the action (e.g., where the ad 1314 ispositioned). In another example, an ad location near the fifty yard linein the football game may be valued or priced higher than an ad locationnear the thirty yard line. Additionally or instead, the value may differbased on the size of the ad location. For example, a larger ad size maycommand a higher price than a smaller ad size at the same ad location.However, a larger ad size in a less desirable location (e.g., off to aside) may not be valued as high as a smaller ad size in a more prominentlocation (e.g., center of the display screen).

In FIG. 14, a display screen 1430 is shown wherein embedded ads 1434,1436, and 1438 are superimposed over the top of the real banner ads thatexisted in the un-enhanced video signal. The enhanced video signalcompletely obscures the real banner ads present at the sports arena andappears to the viewer as though the superimposed embedded ads 1434,1436, and 1438 are inherent to the setting and are a natural part of thescene. In this way, local embedded advertising can supercede nationalembedded advertising for different regions of a nationally broadcastprogram. FIG. 14 also illustrates an informational embedded ad 1440placed on the football field 1432 denoting the end zone of the team onoffense. Similarly, the ad locations over the real banner ads maydictate different values (e.g., price for ad location of ad 1436 may behigher or lower than price for ad location of ad 1438).

In FIG. 15, a display screen 1548 is shown wherein interactive playername embedded ads 1550, 1552, 1554 and 1556 are superimposed over thehelmets of the football players in a game. In this instance, theenhanced video signal is not viewed as inherent to the setting and theembedded ads may be used as hot labels to bring up additional playerinformation when accessed. This information can include personal,biographical or statistical data or the like for the accessed player butrequires the user to click on the embedded ad.

FIGS. 16 and 17 are illustrations of scenes showing other embodiments.Each time that an embedded ad appears in a new scene, the embedded admay be a different advertisement. As shown in FIG. 16, scene one 1602,the person 1604 reaches into an open refrigerator 1606 for a soda can1608. The embedded ad 1610 is a “Coke” logo. In FIG. 17, scene two 1712,the same person 1714, while standing beside the closed refrigerator1716, drinks from the soda can 1718. The embedded ad 1720 is now a“Pepsi” logo. Applying embedded ads in this fashion may be done throughpattern recognition and localized superimposition of ads, as willsubsequently be explained.

FIG. 16 also illustrates the addition of an interactive tattoo embeddedad. In scene one 1602 the person 1604 is depicted with tattoo containingnationally recognized logo 1650 superimposed on the persons arm. In thisscene, the tattoo is overlaid with a transparent background to look likea natural part of the scene. Accessing this interactive hot label maylink the viewer to either the home page of the company depicted in thenationally recognized logo or to an address and phone number of a localtattoo shop.

FIG. 18 is a schematic block diagram that generally illustrates anotherembodiment. As shown in FIG. 18, an origination site 1800 may providecontent. Content comprises an un-enhanced video signal 1802. The contentmay be created by a content provider such as a television network,television studio, a live broadcast, a server providing code or datasuch as an Applications Service Provider, an Internet Service Provider,or any other type of content provider that may be capable of sendingcontent to viewers. The origination site 1800 may be capable ofgenerating the content itself, or may simply provide content that issupplied from an outside source 1804 in the form of an un-enhanced videosignal 1806. The origination site 1800 transmits the un-enhanced videosignal 1802 to a server 1808. The address of the un-enhanced videosignal can be provided by the advertiser to the TV broadcaster.

Referring to FIG. 18, the server 1808 interacts with a central computer1810 and applies an embedded ad 1812 to the un-enhanced video signal1802. This embedded ad 1812 may comprise any form of video enhancement,including video sequences, text or graphical images and may be in anyformat or other graphical representation such as bitmap or GIF. Theserver 1808 accesses on or more of the multiple data files 1822contained in a central computer 1810. There may be one data file foreach advertiser or one data file for each advertisement (e.g., embeddedad) and may comprise an embedded ad 1812 that exists as graphicalinformation within the data file. The data file 1822 may also compriseplacement and contour data 1814 which may be obtained by use of existingcomputer recognition, physical manipulation, or other techniques. Byusing placement and contour data, the embedded ad either may be appliedunobtrusively to the video signal, or may appear to be an obviousaddition to the original video signal 1802. Alternatively, the data file1822 may comprise secondary video streams that may be superimposed(overlaid) onto the original un-enhanced video signal 1802.

Referring again to FIG. 18, the central computer 1810 may also contain adatabase 1816, as referred to in the description of FIG. 9, which storeseach advertiser name and additional information. The central computer1810 may also contain another database 1818, as referred to in thedescription of FIG. 10, which stores each advertiser name, associatedattributes as well as a link 1022, 1024 to the embedded ad (asillustrated in FIG. 10) or an embedded ad text file 1026 and 1028. Eachentry for each advertiser in the database 1818 (as previouslyillustrated in FIG. 10) may be linked to the associated advertiser'sdata file 1822 though a link 1820. Again, the databases and the datafiles may be stored in a central computer or on multiple computers whichcan comprise a server located at a head-end or in a set-top box.

Referring to FIG. 18, the server 1808 may incorporate existing patternrecognition software and techniques to provide one way in whichavailable advertising space may be identified. However, identificationof advertising space can be achieved by several methods as may befurther seen with respect to the description of FIG. 25. The patternrecognition software may communicate with the attribute database 1818,stored in the central computer 1810. As previously discussed, theattributes may be actual objects or “blank space” attributes. Thepattern recognition software may search the unenhanced video stream 1802for these actual objects or blank spaces. Once a pattern is identified,the server searches the attribute database 1818 for a matchingattribute. In the event that more than one company has listed theidentified actual object or blank space as an attribute, bidding maystart to choose which advertiser will receive the embedded ad. Adetailed discussion of criteria, process and application of the embeddedad is described in further detail with respect to FIG. 23.

Referring again to FIG. 18, once the server 1808 has applied theembedded ad 1812 which the server accessed from either the data file1822, or the attribute and text-embedded ad database 1818, the outputfrom the server is an enhanced video signal 1824; for example, a videosignal which has one or more embedded ads applied to the video signal.The enhanced video signal 1824 is transmitted via a link 1826 to aset-top box 1828. The link 1826 may comprise a radio frequency link,satellite link, cable connection, or any other type of communication.The set-top box 1828 receives and transmits the enhanced video signal1824 to a display device 1830, such as a computer monitor or TV screen.

FIG. 19 is a schematic block diagram that generally illustrates anotherembodiment. In FIG. 19, the embedded ad is applied by the set-top box,rather than the server. Content comprises an un-enhanced video signal1902. The content may be created by a content provider such as atelevision network, television studio, a live broadcast, a serverproviding code or data such as an Applications Service Provider, anInternet Service Provider, or any other type of content provider thatmay be capable of sending content to viewers. The origination site 1900may be capable of generating the content itself, or may simply providecontent that is supplied from an outside source 1904 in the form of anun-enhanced video signal 1906. The origination site 1900 transmits theun-enhanced video signal 1902 via a link 1926, which again, may comprisea radio frequency link, a satellite link, or other type of link, to theset-top box 1928. The set-top box 1928 accesses the central computer1910 to apply one or more embedded ads to the un-enhanced video signal1902 to produce an enhanced video signal 1924 output. This enhancedvideo signal 1924 is then sent from the set-top box 1928 to a displaydevice 1930, such as a computer monitor or television screen.

FIG. 20 is a flow diagram that illustrates the steps performed inapplying embedded ads to a video stream 2000. As shown in FIG. 20,content is created at step 2002. As previously disclosed, the contentmay be created by a content provider such as a television network,television studio, a live broadcast, a server providing data such as anApplications Service Provider, an Internet Service Provider, or anyother type of content provider that may be capable of sending content toviewers. The process is continued at step 2004 where availableadvertising space is designated by manual methods. Such manual methodsmay comprise having an author, such as a person examining theun-enhanced video signal, manually designate places within the currentunenhanced video signal that are available for advertising.Alternatively, the process may proceed from step 2002, where the contentis created, to step 2006, where the designation of available advertisingspace is automated. Automatic methods of designating advertising spacemay include visual (pattern) recognition or other techniques. Theprocess proceeds to step 2008, where a list of the available advertisingspaces is generated. This list may be generated by a server that alsocontains a list of advertisers and their associated attributes. At step2010, the available advertising space that has been recently designatedis bid upon by the advertisers. At step 2012, embedded ads are placed inthe video while the video content is being shown on a display device,such as a computer monitor or television screen.

Embedded ads may be representative of local or national companies andmay be superimposed on a per-household basis. This individualizedsuperimposition may be facilitated by utilizing affinity data, such asdisclosed in U.S. patent application Ser. No. 10/046,618, filed Oct. 26,2001 entitled “Collection of Affinity Data from Television, Video, orSimilar Transmissions,” which is based upon U.S. Provisional ApplicationSer. No. 60/243,506, filed Oct. 26, 2000 entitled “Collection ofAffinity Data from Television, Video, or Similar Transmission,” and alsodisclosed in U.S. Non-Provisional application Ser. No. 10/099,054, filedMar. 13, 2002 entitled “Affinity Marketing for Interactive MediaSystems,” which is based on U.S. Provisional Application Ser. No.60/275,295, filed Mar. 13, 2001 entitled “Affinity Marking forInteractive Television,” all of which are specifically incorporatedherein by reference for all that they discloses and teach. Embedded adsmay also be superimposed on a house-by-house basis by collecting datareceived from personalized remote controls, as disclosed in U.S. PatentApplication Ser. Nos. 60/227,916 and 09/941,148 referenced above. Inaccordance with the above applications, affinity data providesinformation to a television broadcaster as to whether or not an audienceis actually engaged in the current video presentation, through use of avariety of sensors, and feedback mechanisms as opposed to simply havingthe television on and not watching the television. The personalizedremote is used to collect data regarding what shows are being watched byeach different member of a household. Together, the use of technologyfrom affinity data and personalized remotes may allow for personaltargeting of advertisements. In that regard, television broadcasters mayprovide data to advertisers as to how many households will receive theirad prior to starting the bidding process. This information may be storedwith the attribute data and transmitted to the advertiser when thetelevision broadcaster solicits bids.

FIG. 21 is a flow diagram illustrating the steps for carrying out theinstallation of embedded ads by manual designation of availableadvertising space 2100. The content or video signal is displayed in step2101. In step 2102, the author, previously defined as the personexamining the video signal, examines the video signal. The processproceeds to step 2104 where the author manually designates availableadvertising space within the un-enhanced video signal. That is, theauthor looks for possible places existing in the original unenhancedvideo stream onto which to place advertisements. Such possible placesmay include actual objects, onto which a particular logo of a companymay be applied or blank spaces onto which any advertiser desiring toplace their advertisement on a blank space may utilize. In addition, theauthor may plan the insertion of objects such as billboards, cans, cars,clothing, etc. In that regard, the existing objects in the program maybe changed. For example, the video signal may show an actress driving aChevy. The embedded advertising has the ability to change the Chevy to aFord. Once the author identifies an embedded advertising opportunity,the author designates the identified opportunity and placement spot asbeing available for an embedded advertisement. Proceeding to step 2106,placement and contour data for the identified object, an insertedobject, or blank space, which can act as the available advertisingspace, is generated. The placement/contour data may be obtained by useof existing computer recognition techniques, physical manipulation, orother techniques, and may be generated by a computer located at thehead-end, the set-top box, or other site. Utility of the contour datamay include making the embedded ad appear more naturally in the scene.As an example, if the object to which the embedded ad is applied isround, the embedded ad, as text or graphics or any visual enhancement,may be wrapped around the object's physical shape so that the embeddedad-enhanced object appears to look natural. By generatingplacement/contour data, the available advertising space is designatedand comprises a boundary around the identified object.

Again referring to FIG. 21, in step 2108, the placement/contour data iscombined with the video signal. In step 2110, the combinedplacement/contour data and video signal are accessed by the computer.The location of the computer may be at the head-end (transmission site),retransmission site, the set top box (household), or some other site. Instep 2112, the embedded ad is applied to the unenhanced video signal. Aspreviously disclosed, the embedded ad may be applied by overlaying data,which may contain the embedded ad itself along with placement/contourdata, onto the original un-enhanced video signal. The data may be in theform of a Hypertext Mark-up Language (HTML) overlay page, which ismostly blank except for the embedded ad that the HTML overlay pagecontains. In this way, only the embedded ad is visible as the videosignal flows through the system. The server or set-top box overlays theHTML page onto the video signal, thus applying the embedded ad toselective areas of the video signal by using the coordinates given bythe placement/contour data. The embedded ad may be color-coded orpartially transparent such that the embedded ad appears to be part ofthe original broadcast to the viewer. Alternatively, the embedded ad maybe presented as an obvious addition to the original video signal, as waspreviously illustrated in FIG. 11.

The embedded ad may constitute graphical, image or text information andmay be in the form of an MPEG stream, a bitmap, a JPEG image, etc. Thegraphical, image, or text information of the embedded ad may be storedlocally in the set-top box, at the head-end, at a distribution point ormay be accessed over the Internet or other network. As indicated above,an HTML overlay page may be used to combine the information of theembedded ad with the pre-existing video stream. However, any type ofcomputer-generated bitmap can be used to produce a screen that can becombined with the pre-existing video signal in one or more frames.

In addition, the computer-generated bitmap need not just overlay thepreexisting video image. For example, the combination of the video imageand the embedded ad image may employ arithmetic or logical combinationsof the bitmap information. Such combinations may employ rasteroperations common to windowed graphical user interface environments suchas Microsoft Windows™. These arithmetic and logical combinations allowthe embedded ad to replace or partially replace the pre-existing videoimage with the content of the embedded ad. Half toning and “see-through”embedded ads can also be produced using such techniques. The resultantvideo signal that is displayed to the viewer allows the embedded ad tothen appear as an integral part of the displayed image. Hence, theprocess of superimposing the embedded ad content on the pre-existingvideo content can be accomplished by simply overlaying a computergenerated bitmap on the pre-existing video content, which may alsocomprise bitmap information generated from an MPEG type of signal.Alternatively, the process of superimposing the embedded ad content onthe pre-existing video content can be accomplished by actually combiningthe computer generated bitmap containing the embedded ad content withthe pre-existing video content such as with arithmetic or logicalcombinations, as disclosed above. Further, commonly used color keyingtechniques, known in the television and computer arts and commonlyreferred to as “blue screen” and “chroma-key”, can also be employed. Forexample, the embedded ad content may include specific colors that havebeen designated to not replace the content of the pre-existing videocontent, or specific colors may be employed to define areas ofreplacement or combination. In this fashion, the “transparency” of theembedded ad content can be accomplished using color key techniques.

The process of selecting the embedded ad content can be accomplished invarious ways. For example, the embedded ad content may be accessed overa network connection. This network connection may be based uponpreferences stored in a set-top box or based upon a tag or marker thatis transmitted as part of the video content of the pre-existing videostream. In addition, the network connection may be an upstreamconnection to a server, such as a server located at the head-end or somedistribution point that has stored embedded ad content that can then bedownloaded to the set-top box either through the vertical blankinginterval (VBI) or an out-of-band channel. In addition, the embedded adcontent that is accessed in this fashion may be based upon preferencesthat are stored either at the head-end, an access point or at theset-top box. In addition, the bitmap information of the embedded adcontent may be generated from instructions that are downloaded from anetwork such as the various network connections described above. Theseinstructions may comprise executable code that is capable of generatinggraphics or text at the set-top box level. Again, these can be basedupon preferences.

While the foregoing description may employ references to a set-top box,it is specifically intended that any and all such description shallapply to any receiving unit operable to provide a similar function,including intelligent televisions, multimedia computers, satellitereceivers, etc., and also applies to portable systems that may employwireless networks to receive video transmission and that may employwireless networks to access embedded ad information.

FIG. 22 is a flow diagram illustrating the steps for carrying out theinstallation of embedded ads by automatic designation of availableadvertising space and searching for actual objects 2200. In previousembodiments of the invention, the designation of available advertisingspace was performed by manual methods. Alternatively, the designation ofopportunities for embedded advertising may be accomplishedautomatically. The following steps outline an automated process ofdesignating available advertising opportunities. In step 2201, thecurrent unenhanced video signal is displayed. The process proceeds tostep 2202 where pattern recognition techniques may be applied to thecurrent unenhanced video presentation. These pattern recognitiontechniques may be performed by a computer at a head-end (transmissionsite), retransmission site, the set top box (e.g., household), or someother site. In step 2204, a query is made as to whether a pattern hasbeen recognized. Recognized patterns may include actual objects. If apattern has not been recognized, the process returns to step 2202 andcontinues applying the pattern recognition component to the currentvideo presentation. Returning to step 2204, if some sort of pattern isrecognized, the process proceeds to step 2206, where a query is made asto whether the recognized pattern is an actual object. If the recognizedpattern is not an actual object, the process returns to step 2202 andcontinues employing the pattern recognition component. After the queryat step 2206, if it is determined that the recognized pattern is anactual object, the process proceeds to step 2208 where placement andcontour data is generated, which may include coordinates of therecognized object. As previously disclosed, placement and contour datamay be generated by use of existing computer recognition techniques,physical manipulation, or other techniques. The placement/contour datamay be generated by a computer located at the head-end, the set-top box,or other site. By generating placement/contour data, the availableadvertising space is designated and comprises a boundary around theidentified object. Recognized objects can also be replaced automaticallyby other objects, as described above.

Referring again to FIG. 22, the placement/contour data is combined withthe video signal in step 2210. Thus, the output signal after step 2210is an enhanced video signal. The placement/contour data and videosignal, collectively referred to as the enhanced video signal, areaccessed by the computer in step 2212. As previously disclosed, thelocation of the computer may be at a head-end (transmission site),retransmission site, the set top box (household), or some other site. Instep 2214, the computer overlays a HTML page which is blank except forthe embedded ad, onto the video signal. Thus, one or more embedded adsare applied to the selective areas of the video signal by using thecoordinates given by the placement/contour data.

FIG. 23 is a flow diagram illustrating the steps for carrying out theinstallation of embedded ads by automatic designation of availableadvertising space and searching for blank spaces 2300. The principlesdiscussed in FIG. 23 are similar to the principles discussed in FIG. 22,with the pattern recognition component searching for a blank spaceinstead of an actual object. As previously disclosed in FIG. 22, thedesignation of available advertising space is completed by use ofautomated methods. The process begins at step 2301 where the unenhancedvideo signal is displayed. The process continues at step 2302, wherepattern recognition techniques may be applied to the unenhanced videosignal. Again, the pattern recognition techniques may be performed by acomputer at a head-end (transmission site), retransmission site, the settop box (household), or some other location. At step 2304, a query ismade as to whether a pattern has been recognized. Recognized patternsmay include blank spaces. If a pattern has not been recognized, theprocess returns to step 2304 and continues applying the patternrecognition component to the current video presentation. Returning tostep 2304, if a pattern is recognized, the process proceeds to step2306, where a query is made as to whether the recognized pattern is ablank space. If the recognized pattern is not a blank space, the processreturns to step 2302 and continues employing the pattern recognitioncomponent. After the query at step 2306, if it is determined that therecognized pattern is a blank space, the process proceeds to step 1208where placement and contour data is generated and may includecoordinates of the recognized object. As previously disclosed, placementand contour data may be generated by use of existing computerrecognition techniques, physical manipulation, or other techniques by acomputer at the head-end, set-top box, or other site. By generatingplacement/contour data, the available advertising space is designatedand comprises a boundary around the identified object.

Referring again to FIG. 23, the placement/contour data is combined withthe video signal in step 2310. The resultant enhanced video signal isaccessed by the computer in step 2312. As previously disclosed, thelocation of the computer may be at the head-end (transmission site),retransmission site, the set top box (household), or some otherlocation. At step 2314 the computer overlays a HTML page, which is blankexcept for the embedded ad, onto the video signal. Thus, one or moreembedded ads are applied to the selective areas of the video signal byusing the coordinates given by the placement/contour data.

FIG. 24 is a flow diagram illustrating the steps for carrying out theinstallation of embedded ads, using automatic methods, in a manner thatcombines the processes of looking for an actual object while searchingfor a blank space simultaneously 2400. As shown in FIG. 24, the authorcreates a list of advertisers at step 2401. At step 2402, attributes areentered and assigned to respective advertisers. As previously defined,attributes are any actual object that is associated with the advertiseror a blank space. Actual object attributes and blank space attributesare illustrated in FIGS. 11-15. As previously discussed in FIG. 8, eachadvertiser may be accompanied by a list of additional information.

Referring again to FIG. 24, at step 2404 a database record of eachcompany and their associated attributes, as illustrated in FIG. 10, iscreated. This record is saved to the database at step 2406. At step2408, the video signal, such as a television sitcom, is played. At step2410 a pattern recognition component is activated, which constantlysearches the video content to identify patterns, e.g., actual objects orblank spaces. At step 2412, a decision is made as to whether or not apattern has been recognized. If a pattern has been recognized, theprocess proceeds to step 2414. If a pattern has not been recognized, theprocess returns to step 2410 and continues searching the video signalfor additional patterns. If a pattern has been recognized at step 2414,a decision is made as to what type of pattern is recognized. If theidentified pattern is a blank space, the process proceeds to step 2416.At step 2416, placement and/or contour data is generated which mayinclude coordinates of the identified object. As previously disclosed,the placement and contour data may be generated by use of existingcomputer recognition techniques, physical manipulation, or othertechniques. By generating placement data, available advertising space isdesignated. At step 2418, the database is searched for an appropriateadvertisement. If no appropriate advertisement is found, the processreturns to step 2410 and continues searching the video signal. At step2420, the advertisers with matching attributes bid on the ad and thewinning bidder is chosen.

Referring back to FIGS. 13-15, the pattern recognition component mayrecognize a blank space, such as arena panels, open ice or field, etc.,onto which multiple advertisers may want to advertise (e.g., multipleadvertisers had listed “blank space” as one of their attributes). Anadvertiser may have chosen an attribute such as a hockey game, baseballgame, Friends episode, Seinfeld episode, situation comedy, prime timebroadcast, rerun, etc. to be placed in the bidding. This allows theadvertisers to be placed in the bidding process for areas of specificinterest and allows the broadcasters to target advertisers in placingbids on topics, times, programs, etc. of interest to the advertisers.

Referring again to FIG. 24, at step 2422, Extensible Mark-up Language(XML) code is created and superimposed (e.g., overlaid) onto thedesignated blank space within the unenhanced video signal. The XML codeis superimposed onto the unenhanced video signal by use of theplacement/contour data previously acquired in step 2416. In the finalstep 2424, code is translated, via an Extensible Scripting Language(XSL) parser, from XML into code for Web TV set-top boxes, HTML, JavaScript (JS) (code for Liberate set-top boxes) and/or code for otherset-top boxes.

Referring again to FIG. 24, the process returns to step 2414. If thepattern recognized is an actual object, the process proceeds to step2426. At step 2426, placement and/or contour data is generated. Again,the placement and contour data may be generated by use of existingcomputer recognition techniques, physical manipulation, or othertechniques. At step 2428, the database is searched for a matchingattribute. If no matching attribute is found, the process returns tostep 2410 and the pattern recognition program continues searching thevideo stream. If a matching attribute is found, the process proceeds tostep 2430. At step 2430, the advertisers that have matching attributesare contacted to place a bid. The information regarding the embedded adis provided to the advertisers. The advertisers bid on the embedded adand the winning bid is chosen. If there are no advertisers with matchingattributes, the broadcaster may wish to categorize the opportunity as ablank space, which means that the object will be removed and a new oneinserted. This provides move opportunities to obtain bidders for theembedded advertising opportunities.

The process then proceeds to step 2432, where XML code is created andsuperimposed on the designated actual object or replaces the object. Inthe final step 2434, code is translated, via an XSL parser, from XMLinto code for Web TV set-top boxes, HTML, JS (code for Liberate set-topboxes) and/or code for other set-top boxes.

FIG. 25 is a flow diagram illustrating the steps performed for theinstallation of embedded ads by using tags and markers to describeadvertising space 2500. As mentioned previously, pattern recognitiontechniques may provide a way in which available advertising space may beidentified. However, identification of advertising space can be achievedby several methods. For example, available advertising space within thevideo stream may be defined by use of technology as disclosed in U.S.Application Ser. Nos. 60/268,350 and 10/076,950 referenced above.

Referring again to FIG. 25, content is created at step 2501. Proceedingto step 2502, available advertising space is designated. As discussed inFIGS. 21 and 22, the designation of available advertising space may beachieved by automatic or manual methods. If the available advertisingspace has been designated by automatic methods, it can then be checkedand edited by manual methods. The process proceeds to step 2504 wheretags are generated as disclosed in the above references. These tagsdescribe the advertising space that is designated in step 2502. Aspreviously disclosed, the advertising space may be an actual object or ablank space, and is referred to as an attribute. The tags define thelocation and perhaps contour of the identified object. That is, the tagsmay comprise a location or several defining locations, perhaps in x andy coordinates, and a contour of the object or blank space (attribute)identified. The tag may flag the attribute as being a blank space or anactual object. This placement and contour data may also be generated byuse of existing computer recognition techniques, physical manipulation,or other techniques. The process proceeds to step 2506, where the tag isplaced in the video stream.

Referring again to FIG. 25, the process proceeds to step 2508 wheremarkers are generated in a manner disclosed above. These markersindicate the position of a division between each video segment of thevideo stream. That is, the markers indicate the beginning and the end ofeach video segment within the video stream. The process proceeds to step2510, where the markers are inserted in the video stream. This can bedone by the content provider or marketer of the program.

At step 2512, a “moving location” tag may be superimposed into the videostream and that indicates the new location of an object on the screen ina manner such as disclosed in U.S. Provisional Application Ser. No.60/354,745, filed Feb. 6, 2002 entitled “TRACKING MOVING OBJECTS ONVIDEO WITH INTERACTIVE ACCESS POINTS”, which is specificallyincorporated herein by reference for all that it discloses and teaches.

For example, an identified object, which may be refrigerator, may moveslightly from one frame to the next. Such a tag would account for therefrigerator's new location and/or orientation, and this tag maycomprise new x and y coordinates and contour data of the refrigerator.The tags and markers may be sent during the video's vertical blankinginterval (VBI). Any known standard for transmitting enhanced data,including advanced television enhancement forum (ATVEF), may be used.References made herein to the ATVEF specification are made forillustrative purposes only, and such references should not be construedas an endorsement, in any manner, of the ATVEF specification.

Referring again to FIG. 25, the process proceeds to step 2514, where thevideo signal, which has been enhanced with tags and markers, istransmitted to a broadcaster or to the head-end where the tag is read.Alternatively, the broadcaster could read the tag, if the advertisementto be superimposed is a national advertisement. Referring again to step2514, the head-end reads the tag marking the location of an object orblank space and providing other information such as object or blankspace profile, size, type, or other information. The process thenproceeds to step 2516 where the head-end queries a list of advertisers,as referred to in FIG. 3. A query is made as to whether the listcontains an advertiser that has listed the identified object or blankspace as one of the advertiser's attributes. At step 2518, advertisersthat have matching attributes are selected for bidding and the owningbidder is selected. The embedded ad is consequently superimposed intothe video stream at head-end at step 2520. The display size of theembedded ad may be determined by the coordinate and contour dataprovided by the tag.

Various embodiments of the present invention therefore provide embeddedadvertising that can be bid upon by advertisers and implemented locally,or even on a household basis, which is the ultimate form of localizedadvertising. Localization of advertisements can be extremely effectiveespecially when combined with personal preference and demographic data.Furthermore, by adding embedded ads to video content in apost-production environment, local advertising can be provided in asimple and inexpensive manner. Targeted advertising in this manner canbe very valuable to prospective advertisers, and advertisers will bewilling to bid high prices for such advertising. As such, revenue ofbroadcasters may be increased. Embedded ads may be unobtrusive,non-intrusive, and do not compromise size, color, or continuity of thevideo content. Embedded ads that are interactive may be activated byseveral methods, including clicking on the embedded ad with a hand-helddevice or remote, moving a pointing device over the embedded ad, orother methods. Embedded ads can transfer information back to thehand-held device for display/interaction, perhaps by a wireless orinfrared link, send information back to a server for data aggregation,or perform any number of other actions. Further, a personalizedhand-held device can be used in accordance with the present inventionthat allows the system to provide personalized information and allowsseveral people to interact with the same program simultaneously. Suchprinciples are disclosed in U.S. Non-Provisional application Ser. No.09/941,148, filed Aug. 27, 2001 entitled “Personalized Remote Control,”which is based upon U.S. Provisional Application Ser. No. 60/227,916,filed Aug. 25, 2000 entitled “Personalized Remote Control,” both ofwhich are specifically incorporated herein by reference for all thatthey disclose and teach.

Embedded ads encourage impulse buying of products or services whichencourages advertisers to bid on embedded advertising. Embedded ads forproducts and services may be directly related to the content beingwatched or may provide additional information for the purchase of itemsthat have no relevance to the video content. Embedded ads can betailored individual one-on-one advertising, allow the viewer to vote,perform other interactive responses, or facilitate access to theworld-wide-web. Embedded ads can display information in a number ofdifferent ways, such as on a relevant item, on a separate page, or on anunused portion of the video presentation. Traditional enhancements, suchas a window or text-box on the side or bottom of the television screenmay appear once the embedded ad is activated. A special small icon,perhaps called a “Buy-Con™,” may appear in the corner of screenindicating to the viewer the availability of embedded ads in the scene.

Embedded ads may be installed automatically by use of existing patternrecognition technology. Patterns that are identified by the software maybe actual objects that exist in the current video stream, or blankspaces that exist in the current video stream. Actual objects or blankspaces may be an appropriate place to superimpose an embedded ad. Thepatterns identified by the pattern recognition software are matched withthe list of attributes to identify an applicable embedded ad, and thepatterns are assigned placement and/or contour data. Theplacement/contour data is then used by a computer to apply the embeddedad in the current video stream. Objects can be replaced with otherobjects having the same or different attributes.

Alternatively, the installation of embedded ads may be achieved bymanually installing embedded ads into the programming content such asdisclosed in U.S. patent application Ser. No. 10/041,881, entitled“Creating On Content Enhancements,” which is based upon U.S. ProvisionalPatent Application Ser. No. 60/243,107, entitled “Creating On ContentEnhancements,” referred to and incorporated above. Initially, manuallyinstalling advertising into the programming content may be done at thehead-end (transmission site), at a retransmission site, or in individualhouseholds according to their demographics. The principles of placingembedded ads on a household-to-household basis are disclosed in U.S.patent application Ser. No. 09/941,148, entitled “Personalized RemoteControl,” which is based upon U.S. Provisional Patent Application Ser.No. 60/227,916, entitled “Personalized Remote Control,” referred to andincorporated above.

Further advantages of the various embodiments disclosed include theability to change embedded ads on a per-scene basis, and the ability toimplement embedded ads automatically or by manual methods. In addition,because embedded ads do not have to be interactive and can exist as partof the video signal. Televisions not enabled for interactivity are stillable to receive the embedded ads. Thus, the application of embedded adsmay not be restricted to video signals, but embedded ads may be appliedto any type of signal and any type of medium.

The various embodiments disclosed provide a system and method thatallows post-production visual alterations in the form of bid-foradvertisements to be added to a video stream. In this manner, broadcastand basic cable channels may provide bid-for embedded advertising tolocal companies in a simple and inexpensive manner. Visual alterations,or embedded ads, may be superimposed into the video stream manually orautomatically. Embedded ads are an effective way to advertise, andembedded ads may be superimposed into the video stream on ahouse-by-house basis. Embedded ads may be personalized and localized.Embedded ads may be applied to any form of viewing material, within anytype of signal or medium. Embedded ads may appear unobtrusively orobviously within the video content, and embedded ads may be eitherinteractive or non-interactive. Embedded ads may change on a per-scenebasis.

In addition, preferences and demographic information that controls thevarious types of ads that are displayed to users can be provided toadvertisers prior to bidding on the embedded advertising. Demographicand preference information that controls the displayed advertising isdisclosed in U.S. patent application Ser. No. 10/688,067, filed Oct. 16,2003, entitled “Demographic/Preference Sniffer,” which is based uponU.S. Provisional Patent Application Ser. No. 60/419,871, filed Oct. 18,2002, entitled “Demographic/Preference Sniffer,” which is specificallyincorporated herein by reference for all that it discloses and teaches.Further, upgraded advertising, such as disclosed in U.S. patentapplication Ser. No. 10/688,075, filed Oct. 17, 2003, entitled “iChooseVideo Advertising,” which is based upon U.S. Provisional ApplicationSer. No. 60/420,110, filed Oct. 18, 2002, entitled “iChoose VideoAdvertising,” which is specifically incorporated herein by reference forall that it discloses and teaches. Information regarding demographicsand preferences that control the display of information can be used tofurther aid the TV advertisers in making bids on embedded advertising.

FIG. 26 with a flow diagram that illustrates the process of auctioningembedded advertising. At step 2602, the broadcaster generates embeddedadvertising opportunities in the manner disclosed above. At step 2604,the broadcaster provides a list of embedded advertising opportunities toadvertisers and advertising specifications including required adinformation necessary to generate the embedded ad and other informationsuch as the expected duration of the ad, a description of the size ofthe ad, viewers who may be viewing the ad, targeted audiences,appearance of the ad and bidding information. There are targetedaudiences, as indicated above, that are based upon either demographicinformation or preference information of viewers. This information canbe provided to the advertiser prior to bidding. The broadcaster thenaccepts bids until the cutoff time at step 2606. At step 2608, thebroadcaster selects the winning bidder. At step 2610, the broadcasterinforms the bidders of the winning bidder and may provide additionalopportunities at that point in time for placing additional embedded adsto both the winning bidder and the losing bidders. At step 2614, thebroadcaster obtains the required ad information from the winning bidderprior to the ad deadline. The broadcaster then generates the embedded adat step 2616. At step 2618, the embedded ad is inserted into thebroadcast in the manner indicated above.

FIG. 27 is a flow diagram illustrating the steps 2700 that discloses theprocess performed by an advertiser in bidding on embedded advertising.At step 2702, the advertiser receives a list of embedded advertisingopportunities from the broadcaster. This list may be created, asdisclosed above, based upon attributes of the business of theadvertiser. For example, an advertiser may be in the business of sellingappliances, in which case, the advertiser may have listed “appliances”as an attribute. If there is an opportunity to place an embedded ad uponan appliance such as a refrigerator in a scene, the attribute“appliance” will be attached to the embedded advertising opportunity. Ofcourse, there may be many other examples of attributes which may be verybroad. As indicated above, the designation of an attribute of “reruns”which can be further limited to a type of programming, or a particularprogram, such as “situation comedy” or “Seinfeld” can be used as aseries of attributes so that an advertiser can obtain relevant embeddedadvertising information.

Referring again to FIG. 27, at step 2706, the advertiser reviews simplevideos, specifications for embedded ads, required ad information andbidding information. In this case, a simple video may be provided alongwith the embedded advertising opportunity which shows an ad embedded ina video clip so that the advertiser can have an idea as to the way inwhich the advertiser's ad may appear when the program is broadcast. Ofcourse, this process applies to prerecorded programming. However, withlive programming, such as a hockey game, video clips of previous gamesshowing embedded ads on the ice and on the boards can be provided as anexample of the manner in which an embedded ad may appear. Again, theadvertiser can determine the manner in which the advertiser's ad mayappear and may influence the amount that the advertiser may wish to bidon such ads.

At step 2708 of FIG. 27, the advertiser then determines if a bid will besubmitted. If not, the process proceeds to step 2710, and the advertiserreviews other embedded advertising opportunities. If the advertiserdetermines it wants to submit a bid, the process proceeds to step 2712to formulate a bidding strategy. Various bidding strategies are known inthe art of bidding and can be used by the advertiser. For example, anadvertiser may wish to submit a very high bid initially in an attempt todrive out other bidders. Of course, the advertiser may determine amaximum amount that the advertiser is willing to pay for the ad prior tosubmitting a first bid. In that regard, if affinity data is providedregarding the success of other embedded ads of the advertiser or thepopularity of the show, those factors may influence the maximum bidamount of the advertiser. At step 2714, the advertiser then submits abid. At step 2716, it is then determined whether the bid that has beensubmitted by the advertiser has won the auction. If it has not, theprocess proceeds to step 2722 where the advertiser may wish to reviewits bidding strategy. At step 2724, the advertiser then determineswhether it wishes to submit another bid. If not, the process proceeds tostep 2710 to review other advertising opportunities. If the advertiserdoes wish to submit another bid, the process returns to step 2714 tosubmit a bid. If the advertiser wins the bid at step 2716, the processproceeds to step 2718 where the advertiser provides the required adinformation to the TV broadcaster. The process then ends at step 2720.

FIG. 28 discloses the process 2800 of modifying bidding strategy basedupon affinity data. Affinity data is data that indicates viewers'responses to a program, an ad or any other information that isdisplayed. Affinity data can be collected by any desired methodincluding surveys, Nielsen rating data, or in accordance with theprocess described in U.S. patent application Ser. No. 10/046,618,entitled “Collection of Affinity Data from Television, Video or SimilarTransmissions,” filed Oct. 26, 2001, by Steven O. Markel, et al.

Referring again to FIG. 28, at step 2802, the advertiser determines thatit has won an advertising bid. At step 2804, the broadcaster generatesan enhanced ad. At step 2806, the enhanced ad is broadcast to viewers.At step 2808, affinity data is collected in accordance with any desiredmethod such as those described above. The affinity data is then gatheredby the advertiser and analyzed at step 2810. At step 2812, theadvertiser may wish to generate a new bidding strategy based upon theaffinity data. For example, if the advertiser is bidding on embedded adsfor the Super Bowl and the game is a very close game in the fourthquarter, the advertiser may wish to bid greater amounts. Of course, thereverse is true, also. For example, if an advertiser is bidding in theSuper Bowl and the score is very lopsided in the fourth quarter, theadvertiser may wish to bid much lower amounts or withdraw a previous bidand pay a penalty fee.

Affinity data can also be used to determine the success of a particularad. For example, affinity data can be collected with regard to theresponse of viewers to an embedded ad that has been displayed previouslyin a program. For example, affinity data can rate a viewer's response toan ad which provides valuable information to the advertiser as towhether the advertiser wishes to bid higher or lower on subsequent adsduring an event such as a football game or hockey game.

Of course, the processes disclosed above are not limited to embedded adsonly. For example, broadcasters may wish to designate certainadvertising time slots as ads which can be bid upon by advertisers. Inthis manner, a broadcaster may wish to maximize its revenue by havingsome time slots available for bidding while maintaining standardpre-purchased advertising time slots so that a guaranteed amount ofincome is provided. In addition, these time slots may have particularrelevance to what is occurring during the program. For example, a timeslot may be designated as a “bid-for” ad after a penalty is assessed ora play is being reviewed in a sporting event. An advertiser may wish topurchase that time slot and run a humorous advertisement showing areferee with thick glasses who has a hard time seeing. In addition, theaffinity data information can be very valuable for these types ofadvertisements to show viewer acceptance or rejection of theadvertisements. By collecting this data, advertisers can determine ifthey would like to run these ads on a more frequent basis and bid higherfor those particular time slots.

FIG. 29 discloses a bidding process 2900 that may be used for liveevents when the broadcaster is willing to extend the auction until atime that is very close to the time in which the embedded ad will bedisplayed. At step 2902, an advertiser receives the criteria foradvertising opportunities for the live event in the various ways asdisclosed above. At step 2904, the advertiser provides the requiredinformation for the embedded ads to the broadcaster. At step 2906, thebroadcaster generates the embedded ads for each advertiser that hassubmitted the required information and stores these embedded ads.Optionally, the broadcaster can provide a sample copy of the embedded adto advertisers. At step 2908, the broadcaster sets the end of auctiontime such that the end of auction time is close to the time for displayof the embedded ad. In this manner, bidding can occur during the processof the live event. For example, bidding may occur during actual play ofa game, such as the Stanley Cup finals, or the Super Bowl. The course ofevents in the game can cause the bids to increase or decrease during thegame as explained above. At step 2910, the bidders submit bids basedupon the progress of the live event. At step 2912, a broadcasterretrieves the embedded ad for the winning bidder. At step 2914, thebroadcaster displays the embedded ad.

FIG. 30 is a flow diagram illustrating the process 3000 of analyzingadvertising opportunities for live events. At step 3002, the broadcastercan review various camera angles and various views of a live event or asimilar event to attempt to determine embedded advertisingopportunities. For example, a broadcaster can view the camera angles orprevious video from a hockey game and determine what embeddedadvertising opportunities exist on the ice and on the boards of the icerink. As indicated above, even though markings may appear on the ice oron the boards, those markings can be removed using either markuplanguage overlay pages or video combination techniques, as disclosedabove. Hence, at step 3004, the potential locations for embedding adsare determined by the broadcaster. Advertisers may also assist thebroadcaster in that evaluation. At step 3006, the broadcaster collectsall of the required ad information from interested advertisers. At step3008, the various embedded ads (e.g., objects) are generated forinsertion in the live program. At step 3010, the broadcaster can thenprovide a sample display of the embedded ads such as an inserted object.For example, during a broadcast of a live football game, an insertedobject may comprise an airplane carrying a banner or a banner that isinserted into the crowd in the stands. Other embedded objects, asdisclosed above, can comprise ads that are placed on the playersthemselves, such as tattoos or labels. At step 3012, the broadcasterobtains suggestions from the advertiser on modifications and additionalembedded advertising opportunities.

FIG. 31 discloses the process 3100 for billing advertisers based upondisplay time of embedded ads. At step 3102, the broadcaster embeds thead in the programming in the manner described above. At step 3104, theembedding software counts the frames in which the full object (e.g., theembedded ad) is displayed on the screen. At step 3106, the total countof frames for each embedded object is provided to the billing software.At step 3108, the billing software generates a total advertising costfor the embedded ad based upon the total count of frames in which theembedded ad appears. Of course, a cap may exist that limits theadvertiser's exposure.

The present invention therefore provides various ways of allowingbroadcasters to auction both embedded ads and normal time slot ads.Auctioning of ads in this fashion allows broadcasters to potentiallyincrease advertising revenue for both prerecorded and live events.Advertisers, on the other hand, also benefit from the bidding process.For example, smaller companies may be able to advertise during largeevents such as the Super Bowl if the event has a smaller interestbecause of a lopsided score. Advertising can constitute targetedadvertising based upon preference and demographic data. In addition,affinity data, that evaluates the effect of the ad, can be used byadvertisers to maximize revenues and determine a bidding strategy.Embedded ads can be generated through markup language overlay pages orthrough combined video techniques. Various strategies for setting upembedded ads by broadcasters can be used including red zone advertisingfor football games, changing all of the embedded ads to a singleadvertiser when the home team scores and providing time slots that arecontent related to a particular advertisement. Attributes of an embeddedadvertising opportunity can be used to select potential advertisers,which also allows potential advertisers to only review embedded ads thatare of interest to the advertiser. Video combination techniques andmarkup language overlay pages can be used to erase existing ads in ascene and insert other ads or insert other objects into the scene. Forexample, if a table exists in a scene, a Coke can or beer can beactually inserted into the scene and an existing item such as an ashtraycan be removed. Similarly, an object such as a car can be erased and adifferent car can-be inserted in a scene to provide the embeddedadvertising opportunity. Various advertising opportunities may beprovided based upon the occurrence of an event in the program, such asthe scoring of a goal or a touchdown. In that case, a trigger may beinserted in the program that triggers the insertion of one or moreembedded ads in the program. In this example, a trigger may be embeddedin the video program which causes the highest bidder who has bid uponthe go-ahead goal on the hockey game or the go-ahead score in a footballgame, etc. to have all the embedded ads changed to that particularadvertiser's embedded ad. Various criteria can be set up such as scoresduring the fourth quarter of a football game, scores during the last fewminutes of a hockey or basketball game, etc. If advertisers wish to dropout of the bidding process, a backup fee for backing out of the processcan be charged to the advertiser. Advertisers can also bid upon puttingembedded ad in certain camera angles. Hence, various methods areproposed for auctioning both embedded advertising and time slotadvertising which provide many advantages to both broadcasters and toadvertisers.

As discussed above, example embodiments allow the determination ofadvertisement opportunities, matching of prospective advertisers withcertain advertisement opportunities, and billing of the advertisementopportunity. In further embodiments, an assigned value of theadvertisement opportunity may be based on location, length of display,or a combination of both. This assigned value may indicate a startingbid price for the advertisement opportunity in an auction typeenvironment or a fixed offer price in fixed price environment (e.g., $1M per minute during the Superbowl).

With reference to FIG. 32, a further example embodiment of a networkarchitecture of an environment 3200 for providing offering and billingadvertisement opportunities is shown. In example embodiments, a centralmanagement system 3202 is communicatively coupled via a network 3204 toan advertiser system 3206 of an advertiser or potential advertiser. Thecentral management system 3202 manages the offering and billing ofadvertisement opportunities within media. The central management system3202 may also offer services such as managing sales of localadvertisement to be inserted into or overlaid on a nationaladvertisement. The central management system 3202 will be discussed inmore detail in connection with FIG. 33 below. While only one advertisersystem 3206 is shown in FIG. 32, any number of advertiser systems 3206of different advertisers may be coupled via the network 3204 to thecentral management system.

The media may comprise one or more of video (e.g., television programs,visual advertisements), audio (e.g., radio programs, audioadvertisements), digital media (e.g., audio, video, data over theInternet) or any combination of these. As such, the network 3204 maycomprise any one or more of a broadcast network, satellite, wide areanetwork (WAN), Internet, cellular network, or any other type ofcommunication network.

In one embodiment, the advertiser system 3206 may comprise anadvertisement manager 3208 and an advertisement database 3210. Theadvertisement manager 3208 allows the advertiser to manage theiradvertisements. For example, the advertisement manager 3208 may trackthe advertisement opportunities the advertiser purchases, theadvertisements to be presented, and accounting (e.g., payment) for theadvertiser. The advertisement database 3210 may store advertisements ofthe advertiser. These advertisements may be sent to the centralmanagement system 3202, a national broadcaster, a local station, ordirectly provided to a user device 3212 of a viewer for embedding intothe media (e.g., by a head-end device). In some embodiments, theadvertisement may be generated by the advertiser system 3206. In otherembodiments, the advertisement may be generated by a third party such asan advertisement agency and stored to the ad database 3210.

The user device 3212 of the viewer is also coupled to the network 3204.The user device 3212 may comprise a set-top box, television, radio,mobile device, desktop computer, or any other communication device thatthe viewer may utilize to receive media transmissions from a broadcastersuch as a national broadcaster, a local station, or the centralmanagement system 3202. In some embodiments, the user device 3212 maycomprise a display module to display information and an audio module tooutput audio to the user. In further embodiments, the user device 3212may comprise an input module to receive user inputs to control orrespond to various media presented on the user device 3212. While onlyone user device 3212 is shown in FIG. 32, any number of user devices3212 may be coupled via the network 3204 to the central managementsystem 3202.

Referring now to FIG. 33, an example block diagram illustratingcomponents of the central management system 3202 are shown. In exampleembodiments, the central management system 3202 comprises a sales engine3302, a billing engine 3304, and a data storage 3306. The sales engine3302 is configured to manage the determination of advertisementopportunities and the offering of these advertisement opportunities forsale or auction. The sales engine 3302 will be discussed in more detailin connection with FIG. 34 below. The billing engine 3304 is configuredto manage valuation of advertisement opportunities and invoicing ofpurchased advertisement opportunities. The billing engine 3304 will bediscussed in more detail in connection with FIG. 35 below. It is notedthat FIG. 33 provides one example of the central management system 3202,and alternative embodiments may comprise more, fewer, or othercomponents to enable the operations described herein.

The data storage 3306 comprises a plurality of databases that are usedby the engines and modules of the central management system 3202. Thesedatabases may include an advertisement database 3308, advertiserdatabase 3310, and a billing database 3312. The advertisement database3308 stores data regarding advertisement opportunities. The data mayinclude attributes, assigned values, identification of winningadvertiser, and a copy of an advertisement of the winning advertiser foreach of the advertisement opportunities.

The advertiser database 3310 stores data associated with the advertiser.As discussed above, the advertiser may register with the centralmanagement system 3202 and provide attributes for their advertisement orattributes for advertisement opportunities for which the advertiserdesires to purchase. These registration and attribute information may bestored in an account associated with the advertiser in the advertiserdatabase 3310. In example of the advertiser database 3310 is shown anddescribed with respect to FIG. 9 above.

The billing database 3312 stores billing and valuation information forthe advertisement opportunities. The billing information may include anaccounting of a number of frames or an amount of time that anadvertisement is displayed in an advertisement opportunity. Thevaluation information may include information used by the billing engine3304 in determining a value to assign to the advertisement opportunity.The valuation information may include algorithms and weights to apply tovarious attributes and factors in determining the value for anadvertisement opportunity as will be discussed in further detail below.

FIG. 34 is a block diagram illustrating an example embodiment of thesales engine 3302. The sales engine 3302 is configured to manage thedetermination of advertisement opportunities and the offering of theseadvertisement opportunities for sale or auction. In example embodiments,the sales engine 3302 comprises an ad identifier module 3402, anadvertiser module 3404, an ad match module 3406, an auction module 3408,and a fixed price module 3410.

The ad identifier module 3402 identifies advertisement opportunities inmedia (e.g., a television show, movie, sporting event). In someembodiments, the ad identifier module 3402 incorporates patternrecognition software and techniques to identify available advertisingspace as discussed above. The pattern recognition software maycommunicate with the attribute database 1818 or the ad database 3308 todetermine attributes to search for in the media. As previouslydiscussed, the attributes may be actual object attributes or “blankspace” attributes. The pattern recognition software may search the media(e.g. a video stream) for these actual objects or blank spaces. Forexample, a pizza box may be identified in the middle of a scene. Theattributes associated with this object (e.g., pizza box) may includeobject type (e.g., box or pizza box), dimensions of the object, locationof the object, scene type, program type (e.g., comedy movie, sportingevent, sitcom), and so forth. In some embodiments, a producer ordistributor of the program or media may manually provide some of theseattributes.

Additionally, the ad identifier module 3402 may generate placement andcontour data that may include coordinates of the recognized object. Aspreviously disclosed, placement and contour data may be generated by useof computer recognition techniques, physical manipulation, or othertechniques. While the pattern recognition process is discussed as beingperformed by the sales engine 3302 of the central management system3202, alternative embodiments may perform this process at a similar (adidentifier) module located at the head-end, transmission site,retransmission site, the set top box (e.g., at viewer location), or someother site. The attributes, placement data, and contour data may bestored to the ad database 3308.

The advertiser module 3404 manages data stored for the potentialadvertisers. Initially, the advertiser may register with the centralmanagement system 3202 via the advertiser module 3404. During orsubsequent to creating an account with the central management system3202, the advertiser provides attributes associated with their line ofbusiness and their advertisement (e.g., pizza, food, take-out),attributes for the advertiser (e.g., pizza company), or attributes foradvertisement opportunities for which the advertiser is interested inpurchasing (e.g., sporting event, sitcom, kitchen scene) via theadvertiser module 3403. The registration and attribute information maybe stored in their account in the advertiser database 3310 by theadvertiser module 3404.

Once a pattern is identified for the advertisement opportunity andcorresponding advertisement attributes determined, the ad match module3406 may search the advertiser database 3310 for advertisers that havematching attributes. Advertisers that have one or more matchingattributes to those indicated by the advertisement opportunity may beoffered the advertisement opportunity. In example embodiments, theadvertiser may be contacted by the central management system 3202offering the advertisement opportunity. In some embodiments, theadvertisement opportunity may go to the highest bidder in an auction. Inother embodiments, the advertisement opportunity may go to theadvertiser that responds the fastest to a fixed price offer.

In the event that more than one advertiser (e.g., company) has listedattributes that match attributes of the identified actual object orblank, an auction may be presented by the auction module 3408. Theauction allows the advertisers to be placed in the bidding process forareas of specific interest and allows the broadcasters to targetadvertisers in placing bids on topics, times, programs, etc. of interestto the advertisers. The advertisers that have matching attributes arecontacted to place a bid by the auction module 3408. The informationregarding the advertisement opportunity is provided to the advertisers.The information may include a starting bid which represents an assignedvalue to the advertisement opportunity. The assigned value will bediscussed in more detail below. If there are no advertisers withmatching attributes or none of the matching advertisers decide to bid inthe auction, the advertisement opportunity may be recategorized as ablank space, thus opening up the bidding process to more advertisers(e.g., advertisers that have indicated a blank space attribute). Thisprovides more opportunities to obtain bidders for the advertisementopportunity.

The fixed price module 3410 manages fixed price sales of advertisementopportunities. In some situations, the broadcaster or other entity maydecide not to use an auction process to sell the advertisementopportunity. Instead, the advertisement opportunity is offered at afixed price. Advertisers with matching attributes may still becontacted. However, the information provided to the potential advertiserwill include the fixed assigned value (e.g., fixed price) that theadvertisement opportunity will be sold for. The advertisementopportunity may then be sold to the first advertiser to respond to theoffer.

FIG. 35 is a block diagram illustrating an example embodiment of abilling engine 3304. The billing engine 3304 is configured to managevaluation of each advertisement opportunity and invoicing of purchasedadvertisement opportunities. The value assigned to the advertisementopportunity may be used by the sales engine 3302 in presenting the offerto potential advertisers. Accordingly, the billing engine 3304 comprisesan attribute module 3502, an analysis module 3504, and an accountingmodule 3506.

The attribute module 3502 manages use of attributes in the billingengine 3304. The attribute module 3502 may retrieve attributes from thead database 3308. As discussed above, the attributes are used toidentify the advertisement opportunity and prospective advertisers.These attributes describe aspects of the advertisement opportunity suchas location in a scene to be displayed on the screen, size of an objectwhere the advertisement opportunity will be presented, and type ofevent/program (e.g., sporting event, sitcom, comedic movie). Theattributes may also indicate, or be used by the attribute module 2502 todetermine a level of activity with respect to the advertisementopportunity. For example with respect to the pizza box scene discussedabove, the pizza box is located in the middle of the scene and isprominent in the scene since the actor is reaching into the box to takeout a slice of pizza. In this example, the level of activity assigned tothe pizza box would be high as opposed to an object that is found in acorner of the scene away from the actor. For example, a level ofactivity value of “10” may be assigned to the pizza box, while a levelof activity value of “2” may be assigned to a blank space on a side ofthe scene. It is noted that the value of an ad placement for a givenadvertiser may also depend on a genre of the visual location. Forexample, it may not make sense to advertise a breakfast cereal on apizza box. Also, the attribute module 3502 may consider visibility ofthe placement in terms of clarity of, for example, an embedded logo. Forinstance, a straight on shot of the pizza box lid may be more valuablethan a shot where the viewer only sees the lid at an angle. Both ofthese instances would rely on some evaluation of the visual impact ofthe ad placement. By assigning level of activity values to theadvertisement opportunity, the level of activity values may be combinedwith other assigned values to determine an overall value (e.g., monetaryvalue) for the advertisement opportunity as well being used to rank theadvertisement opportunities.

In another example discussed above, different ad locations on a playingfield of a sporting event may be valued differently. For example, in afootball game where action is centered around the fifty yard zone, thatad location has a higher level of activity than an ad location furtheraway from the action (e.g., near an end zone). The higher level ofactivity correlates to higher audience focus on the fifty yard zone andaway from the end zone. Therefore, the fifty yard zone, in this example,will have a higher level of activity assigned to it.

The attributes may also include attributes directed to a plot line inthe media. For example, the advertisement opportunity may be identifiedas occurring during a slow portion of an event/program (e.g., movie ortelevision show). Alternatively, the advertisement opportunity may occurduring a critical scene in the program (e.g., conclusion, cliff hanger,twist in plot line). These attributes will be factors in determining thevalue assigned to the advertisement opportunity wherein advertisementopportunities in a critical scene is valued higher than advertisementopportunities in a slow portion of the event or program. For example, aplot line value of “10” may be assigned to a murder scene versus a plotline value of “2” for someone walking down a hallway or a plot linevalue of “7” of ran intense interrogation scene. Besides the type ofscene, the value may be based on a relevance of the scene to the plotline. For instance, the interrogation scene may be more valuable thanthe murder scene. Perhaps only a victim was shown in the murder scene,but in the detective scene we see where an investigator tricks a bad guyinto revealing some important fact. Equally, may be a lab scene beforethe interrogation scene is more important as that is where acriminologist deduces who the murderer is from the evidence. These plotline values may be combined with other assigned values to determine anoverall value (e.g., monetary value) for the advertisement opportunity.

The analysis module 3504 assigns the value to the advertisementopportunity. In one embodiment, the analysis module 3504 may calculatethe value for the advertisement opportunity. Accordingly, the analysismodule 3504 takes as input the various attributes including attributevalues such as the level of activity value and the plot line value forthe advertisement opportunity and determines a total value or a range ofvalues for the advertisement opportunity. For instance, a premium price(e.g., total value) may be determined for the advertisement opportunityon the pizza box example discussed above due to its high level ofactivity, prominent location (e.g., near center of the scene), orplacement clarity of an ad (e.g., viewing angle). The analysis module3504 may also factor in plot line attributes (e.g., plot lineattributes). For example, if the advertisement opportunity is during aslow portion of the program, the value will be lower than anadvertisement opportunity that occurs during a critical portion of theprogram (e.g., based on a pace of a storyline). Other attributes andattribute values may be based on a mood of a scene (e.g., romantic,comedic, dramatic, action packed) or importance of characters in thescene (e.g., Brad Pitt talking to George Clooney as main characters maybe more valuable than a setup set with two lesser known actors talking).The premium value or price may be in the form of a higher starting bidprice or a higher fixed price as compared to a price for an ad locationlocated in an area with lower visibility, lower activity, or smallersize.

The analysis module 3504 may need to balance and weigh the variousattributes in determining the value for the advertisement opportunity.For example, a larger ad size may command a higher price than a smallerad size at the same ad location. However, a larger ad size in a lessdesirable location (e.g., off to a side) may not have as high a value asa smaller ad size in a more prominent location (e.g., center of thedisplay screen or closer to higher levels of activity) or during a morecritical point in a plot line. As such, the analysis module 3504 mayapply various weights to the attributes. These weights or weightingalgorithms may be retrieved from the billing database 3312. For example,one algorithm may weight level of activity higher than plot lineattributes, which in turn, may be weighted higher than size of theadvertisement opportunity. These weights may be adjusted by a user ofthe central management system 3202. In another example, the (total)value for the advertisement opportunity may be based on a match betweenad attributes (e.g., mood, pace, actors) and the scene or show in whichthe ad will appear. For instance, an ad with Drew Barrymore appearingwithin a show with Drew Barrymore may be more valuable than an ad withGlen Close appearing in a show with Drew Barrymore, but does not containGlen Close.

Potentially, the analysis module 3504 may evaluate an impact of a givenad appearing between other ads. For example, if a first ad is a pizzaad, it makes more sense to put a soft drink or beer ad after the firstad than another pizza (or other fast food) ad. So, the value may bebased on matching the ad to some contextual factors in which the ad willappear. This is analogous to the viewer affinity discussed earlier inthe application; just that it is a “content affinity” instead.

In an alternative embodiment, the analysis module 2504 may provide asuggestion for a value, a range of values, or a comparison of thevarious attributes to a user of the central management system 3202. Theuser may then manually assign a value to the advertisement opportunitybased on the information provided by the analysis module 2504. The valueassigned to the advertisement opportunity may then be used by theauction module 3408 or the fixed price module 3410 in offering theadvertisement opportunity for sale.

The accounting module 3506 manages the invoicing and collection ofpayments. In some embodiments, the billing will be based on the finalauction price or the fixed price agreed upon using the sales engine3302. In other embodiments, the billing may be based, at least in part,on a length of time. For example, a total count of frames for eachembedded ad (e.g., advertisement opportunity with an advertisementembedded therein) is determined by the accounting module 3506. In somecases, the length may not be determinable before the actual display ofthe embedded ad. For example, a play during a sporting event may bedelayed or take longer than usual thus resulting in the embedded adbeing displayed for a longer period of time. Furthermore, the billingmay be based on a combination of both auctions/fixed price and length oftime. For example, the advertiser may agree to pay a fixed price of$20,000 for every 10 seconds their ad is shown, or the advertiser mayhave won an auction where the advertiser agreed to pay $25,000 for every10 seconds of display time for their ad. In these examples, theaccounting module 3506 generates a total advertising cost for theembedded ad based upon length of time (e.g., frame count) in which theembedded ad appears and apply that length of time to the auction orfixed price. In some cases, a cap may exist that limits the advertiser'sexposure.

FIG. 36 is a flow diagram of an example high-level method 3600 formanaging offers and billing of advertisement. In operation 3602, anadvertisement opportunity is detected in a piece of media. Attributesfor the advertisement opportunity is determined in operation 3604. Inexample embodiments, the ad identifier module 3402 identifiesadvertisement opportunities in the media using, for example, patternrecognition software. Additionally, the pattern recognition software maydetermine attributes to search for in the media to identify objectswhere advertisement opportunities may be embedded. These objects may beidentified based on actual object attributes or “blank space”attributes. The attributes used to determine the advertisementopportunities may be associated with advertisement opportunities.

In operation 3606, an assigned value (or range of values) is determinedfor the advertisement opportunity. In one embodiment, the analysismodule 3504 calculates the value based on the various attributesdetermined for the advertisement opportunity. The analysis module 3504may also factor in plot line attributes, visibility attributes, level ofactivity attributes, size attributes, or any other attribute desired tobe used into the determination of the value. The analysis module 3504may also balance and weigh the various attributes in determining thevalue. The weights may be adjusted by a user of the central managementsystem 3202. The resulting value (or range of values) may be thesuggested starting bid price in an auction or a fixed price in an offerto sell the advertisement opportunity. Alternatively, the value (orrange of values) may be a suggestion to a user of the central managementsystem 3202, and the user may then manually assign a value to theadvertisement opportunity.

Potential advertisers are determined in operation 3608. Using theattributes identified for the advertisement opportunity in operation3604, the ad match module 3406 may search the advertiser database 3310for the potential advertisers that have matching attributes. Thematching attributes may be directed to characteristics of an object orspace detected as an advertisement opportunity and may, in oneembodiment, include an attribute of “blank space.” Since these potentialadvertisers have indicated interest in advertisement opportunitieshaving one or more of the identified attributes, these potentialadvertisers are more likely to respond to an offer for sale of theadvertisement opportunity.

In operation 3610, an offer of the advertisement opportunity ispresented to the potential advertises determined in operation 3610. Inexample embodiments, the potential advertiser will be contacted by thecentral management system 3202 offering the advertisement opportunity.In some cases, an auction may be presented by the auction module 3408.The auction allows the potential advertisers to be placed in a biddingprocess for the advertisement opportunity. The potential advertisersthat have matching attributes are contacted to place a bid by theauction module 3408. The information regarding the advertisementopportunity is provided to the potential advertisers. The informationmay include a starting bid which represents the assigned valuedetermined in operation 3606.

In other cases, the advertisement opportunity may be offered at a fixedprice. The fixed price module 3410 contacts the potential advertiserswith matching attributes with the offer of the fixed price advertisementopportunity, which may include the fixed assigned value determined inoperation 3606. The advertisement opportunity may then be sold to thefirst advertiser to respond to the offer.

The winner (of an auction) or buyer (of a fixed price offer) will havetheir advertisement embedded into the advertisement opportunity inoperation 3612. Various methods of embedding advertisement have beendiscussed above and any one of these methods may be used herein.

In operation 3614, the advertiser is billed for the advertisementplacement. In some embodiments, the billing will be based on the finalauction price or the fixed price agreed upon using the sales engine3302. In other embodiments, the billing may be based, at least in part,on a length of time the advertisement is displayed to viewers. Forexample, a total count of frames for each embedded advertisement (e.g.,advertisement opportunity with an advertisement embedded therein) isdetermined by the accounting module 3506. In further cases, the billingmay be based on a combination of both auctions/fixed price and length oftime. For example, the advertiser may agree to pay a fixed price of$20,000 for every 10 seconds their advertisement is shown, or theadvertiser may have won an auction where the advertiser agreed to pay$25,000 for every 10 seconds of display time for their advertisement. Inthese examples, the accounting module 3506 generates a total advertisingcost for the embedded ad based upon length of time (e.g., frame count)in which the embedded advertisement appears and apply that length oftime to the auction or fixed price. In some cases, a cap may exist thatlimits the advertiser's exposure.

It is noted that while a particular order to the operations is shown inFIG. 36, the operations may be performed in a different order. Forexample, the advertiser may be billed (operation 3614) prior to theembedding of their advertisement (operation 3612). In another example,the assigned value may not be determined (operation 3606) until afterthe potential advertisers are determined (operation 3608).

Modules, Components, and Logic

Additionally, certain embodiments described herein may be implemented aslogic or a number of modules, engines, components, or mechanisms. Amodule, engine, logic, component, or mechanism (collectively referred toas a “module”) may be a tangible unit capable of performing certainoperations and configured or arranged in a certain manner. In certainexample embodiments, one or more computer systems (e.g., a standalone,client, or server computer system) or one or more components of acomputer system (e.g., a processor or a group of processors) may beconfigured by software (e.g., an application or application portion) orfirmware (note that software and firmware can generally be usedinterchangeably herein as is known by a skilled artisan) as a modulethat operates to perform certain operations described herein.

In various embodiments, a module may be implemented mechanically orelectronically. For example, a module may comprise dedicated circuitryor logic that is permanently configured (e.g., within a special-purposeprocessor, application specific integrated circuit (ASIC), or array) toperform certain operations. A module may also comprise programmablelogic or circuitry (e.g., as encompassed within a general-purposeprocessor or other programmable processor) that is temporarilyconfigured by software or firmware to perform certain operations asdescribed herein. It will be appreciated that a decision to implement amodule mechanically, in dedicated and permanently configured circuitry,or in temporarily configured circuitry (e.g., configured by software)may be driven by, for example, cost, time, energy-usage, and packagesize considerations.

Accordingly, the term “module” should be understood to encompass atangible entity, be that an entity that is physically constructed,permanently configured (e.g., hardwired), or temporarily configured(e.g., programmed) to operate in a certain manner or to perform certainoperations described herein. Considering embodiments in which modules orcomponents are temporarily configured (e.g., programmed), each of themodules or components need not be configured or instantiated at any oneinstance in time. For example, where the modules or components comprisea general-purpose processor configured using software, thegeneral-purpose processor may be configured as respective differentmodules at different times. Software may accordingly configure theprocessor to constitute a particular module at one instance of time andto constitute a different module at a different instance of time. Theprocessor may be temporary configured or permanently configured toperform relevant operations. Whether temporarily configured orpermanently configured, such processors may constituteprocessor-implemented modules that operate to perform one or moreoperations or functions.

Modules can provide information to, and receive information from, othermodules. Accordingly, the described modules may be regarded as beingcommunicatively coupled. Where multiples of such modules existcontemporaneously, communications may be achieved through signaltransmission (e.g., over appropriate circuits and buses) that connectthe modules. In embodiments in which multiple modules are configured orinstantiated at different times, communications between such modules maybe achieved, for example, through the storage and retrieval ofinformation in memory structures to which the multiple modules haveaccess. For example, one module may perform an operation and store theoutput of that operation in a memory device to which it iscommunicatively coupled. A further module may then, at a later time,access the memory device to retrieve and process the stored output.Modules may also initiate communications with input or output devicesand can operate on a resource (e.g., a collection of information).

Example Machine Architecture and Machine-Readable Medium

With reference to FIG. 37, an example embodiment extends to a machine inthe example form of a computer system 3700 within which instructions forcausing the machine to perform any one or more of the methodologiesdiscussed herein may be executed. In alternative example embodiments,the machine operates as a standalone device or may be connected (e.g.,networked) to other machines. In a networked deployment, the machine mayoperate in the capacity of a server or a client machine in server-clientnetwork environment, or as a peer machine in a peer-to-peer (ordistributed) network environment. The machine may be a personal computer(PC), a tablet PC, a set-top box (STB), a Personal Digital Assistant(PDA), a cellular telephone, a web appliance, a network router, a switchor bridge, or any machine capable of executing instructions (sequentialor otherwise) that specify actions to be taken by that machine. Further,while only a single machine is illustrated, the term “machine” shallalso be taken to include any collection of machines that individually orjointly execute a set (or multiple sets) of instructions to perform anyone or more of the methodologies discussed herein.

The example computer system 3700 may include a processor 3702 (e.g., acentral processing unit (CPU), a graphics processing unit (GPU) orboth), a main memory 3704 and a static memory 3706, which communicatewith each other via a bus 3708. The computer system 3700 may furtherinclude a video display unit 3710 (e.g., a liquid crystal display (LCD)or a cathode ray tube (CRT)). In example embodiments, the computersystem 3700 also includes one or more of an alpha-numeric input device3712 (e.g., a keyboard), a user interface (UI) navigation device orcursor control device 3714 (e.g., a mouse), a disk drive unit 3716, asignal generation device 3718 (e.g., a speaker), and a network interfacedevice 3720.

Machine-Readable Storage Medium

The disk drive unit 3716 includes a machine-readable storage medium 3722on which is stored one or more sets of instructions 3724 and datastructures (e.g., software instructions) embodying or used by any one ormore of the methodologies or functions described herein. Theinstructions 3724 may also reside, completely or at least partially,within the main memory 3704 or within the processor 3702 duringexecution thereof by the computer system 3700, with the main memory 3704and the processor 3702 also constituting machine-readable media.

While the machine-readable storage medium 3722 is shown in an exampleembodiment to be a single medium, the term “machine-readable storagemedium” may include a single medium or multiple media (e.g., acentralized or distributed database, or associated caches and servers)that store the one or more instructions. The term “machine-readablemedium” shall also be taken to include any tangible medium that iscapable of storing, encoding, or carrying instructions for execution bythe machine and that cause the machine to perform any one or more of themethodologies of embodiments of the present invention, or that iscapable of storing, encoding, or carrying data structures used by orassociated with such instructions. The term “machine-readable storagemedium” shall accordingly be taken to include, but not be limited to,solid-state memories and optical and magnetic media. Specific examplesof machine-readable storage media include non-volatile memory, includingby way of example semiconductor memory devices (e.g., ErasableProgrammable Read-Only Memory (EPROM), Electrically ErasableProgrammable Read-Only Memory (EEPROM), and flash memory devices);magnetic disks such as internal hard disks and removable disks;magneto-optical disks; and CD-ROM and DVD-ROM disks.

Transmission Medium

The instructions 3724 may further be transmitted or received over acommunications network 3726 using a transmission medium via the networkinterface device 3720 and utilizing any one of a number of well-knowntransfer protocols (e.g., HTTP). Examples of communication networksinclude a local area network (LAN), a wide area network (WAN), theInternet, mobile telephone networks, POTS networks, and wireless datanetworks (e.g., WiFi and WiMax networks). The term “transmission medium”shall be taken to include any intangible medium that is capable ofstoring, encoding, or carrying instructions for execution by themachine, and includes digital or analog communications signals or otherintangible medium to facilitate communication of such software.

Although an overview of the inventive subject matter has been describedwith reference to specific example embodiments, various modificationsand changes may be made to these embodiments without departing from thebroader spirit and scope of embodiments of the present invention. Suchembodiments of the inventive subject matter may be referred to herein,individually or collectively, by the term “invention” merely forconvenience and without intending to voluntarily limit the scope of thisapplication to any single invention or inventive concept if more thanone is, in fact, disclosed.

The embodiments illustrated herein are described in sufficient detail toenable those skilled in the art to practice the teachings disclosed.Other embodiments may be used and derived therefrom, such thatstructural and logical substitutions and changes may be made withoutdeparting from the scope of this disclosure. The Detailed Description,therefore, is not to be taken in a limiting sense, and the scope ofvarious embodiments is defined only by the appended claims, along withthe full range of equivalents to which such claims are entitled.

As used herein, the term “or” may be construed in either an inclusive orexclusive sense. Additionally, although various example embodimentsdiscussed below focus on a insertion of localized advertisement datainto a national advertisement, the embodiments are given merely forclarity in disclosure. Thus, any insertion of localized advertisementdata into any type of media may employ various embodiments of the systemand method described herein and may be considered as being within ascope of example embodiments. Each of a variety of example embodimentsis discussed in detail below.

Moreover, plural instances may be provided for resources, operations, orstructures described herein as a single instance. Additionally,boundaries between various resources, operations, modules, engines, anddata stores are somewhat arbitrary, and particular operations areillustrated in a context of specific illustrative configurations. Otherallocations of functionality are envisioned and may fall within a scopeof various embodiments of the present invention. In general, structuresand functionality presented as separate resources in the exampleconfigurations may be implemented as a combined structure or resource.Similarly, structures and functionality presented as a single resourcemay be implemented as separate resources. These and other variations,modifications, additions, and improvements fall within a scope ofembodiments of the present invention as represented by the appendedclaims. The specification and drawings are, accordingly, to be regardedin an illustrative rather than a restrictive sense.

What is claimed is:
 1. A method comprising: accessing, using one or moreprocessors of a machine, media to be communicated to a user device of aviewer; accessing a communicatively coupled database that storesattributes to search for in the media, the attributes to search for inthe media comprising attributes of an object of an advertisementopportunity in which an advertisement is to be embedded; performingpattern recognition, using pattern recognition software, to identify theobject in the media that includes the attributes from the database;retrieving, from a second database, valuation information for theadvertisement opportunity, the valuation information indicating a valuefor the advertisement opportunity based on one or more of the attributesof the advertisement, including at least a plot line attribute and anactivity attribute, wherein the plot line attribute is a function of acriticality of an associated scene in a plot line comprising a pluralityof scenes, wherein the value is higher for critical scenes compared tonon-critical scenes, and wherein the activity attribute is associatedwith a determined level of activity of the scene; determining, using oneor more processors of the machine, the value for the advertisementopportunity based on the; offering the advertisement opportunity to apotential advertiser based on the value determined using the attributesof the advertisement opportunity; in response to the potentialadvertiser accepting the offering, graphically embedding anadvertisement associated with the potential advertiser into a displayarea of the object based on placement and contour data for the objectgenerated by the pattern recognition software; and transmitting themedia through a media distribution network to the user device of theviewer.
 2. The method of claim 1, wherein the attributes comprise aphysical location of the advertisement opportunity in a scene of themedia, the value assigned to the advertisement opportunity being higherbased on the location being a prominent location in the scene.
 3. Themethod of claim 2, wherein the prominent location within the scenecomprises an area where a level of activity in the scene is higher thanother areas of the scene.
 4. The method of claim 2, wherein theprominent location comprises an area near a center of the scene, thecenter being displayable at a center of a display screen of the userdevice.
 5. The method of claim 1, further comprising: accessing anadvertiser database containing attributes associated with potentialadvertisers; and determining the potential advertiser having attributesthat match the attributes of the advertisement opportunity.
 6. Themethod of claim 1, further comprising billing an advertiser thatpurchases the advertisement opportunity based on the value assigned tothe advertisement opportunity.
 7. The method of claim 6, wherein thebilling comprises: determining a length of time an advertisement isdisplayed; and factoring the length of time into a calculation of afinal invoice.
 8. The method of claim 1, wherein the identifying theobject of the advertisement opportunity and determining the attributesoccurs at the user device of the viewer.
 9. The method of claim 1,wherein the identifying the object of the advertisement opportunity anddetermining of the attributes occurs at a server of a central managementsystem.
 10. The method of claim 1, wherein the determining the valuefurther comprises including a weighted value for a plot line attribute.11. The method of claim 1, wherein the graphically embeddedadvertisement comprises an interactive advertisement, the interactiveadvertisement being selectable by the viewer to provide additionalinformation related to the interactive advertisement.
 12. The method ofclaim 1, wherein the value for the advertisement opportunity is based onboth a first level of activity in the scene relative to one or moreother scenes, and a second level of activity in a first area of thescene relative to one or more areas of the scene.
 13. The method ofclaim 12, wherein the plot line attribute indicates a relevance of thescene to a plot line.
 14. The method of claim 1, wherein the plot lineattribute indicates an importance of characters in the scene.
 15. Themethod of claim 14, wherein the plot line attribute indicates a mood ofthe associated scene as being comedy.
 16. A non-transitorymachine-readable medium storing instructions which, when executed by theat least one processor of a machine, cause the machine to performoperations comprising: accessing media to be communicated to a userdevice of a viewer; accessing a communicatively coupled database thatstores attributes to search for in the media, the attributes to searchfor in the media comprising attributes of an object of an advertisementopportunity in which an advertisement is to be embedded; performingpattern recognition, using pattern recognition software, to identify theobject in the media that includes the attributes from the database;retrieving, from a second database, valuation information for theadvertisement opportunity, the valuation information indicating a valuefor the advertisement opportunity based on one or more of the attributesof the advertisement, including at least a plot line attribute and anactivity attribute, wherein the plot line attribute is a function of acriticality of an associated scene in a plot line comprising a pluralityof scenes, wherein the value is higher for critical scenes compared tonon-critical scenes, and wherein the activity attribute is associatedwith a determined level of activity of the scene; determining the valuefor the advertisement opportunity based on the valuation information;offering the advertisement opportunity to a potential advertiser basedon the value determined using the attributes of the advertisementopportunity; in response to the potential advertiser accepting theoffering, graphically embedding an advertisement associated with thepotential advertiser into a display area of the object based onplacement and contour data for the object generated by the patternrecognition software; and transmitting the media through a mediadistribution network to the use device of the viewer.
 17. Thenon-transitory machine-readable medium of claim 16, wherein theattributes comprise a physical location of the advertisement opportunityin a scene of the media, the value assigned to the advertisementopportunity being higher based on the location being a prominentlocation in the scene.
 18. The non-transitory machine-readable medium ofclaim 16, wherein the value comprises a starting bid price in an auctionfor the advertisement opportunity or a fixed price in a fixed priceoffer for the advertisement opportunity, determining the potentialadvertiser having attributes that match the attributes of theadvertisement opportunity.
 19. The non-transitory machine-readablemedium of claim 16, wherein the operations further comprise: accessingan advertiser database containing attributes associated with potentialadvertisers; and determining the potential advertiser having attributesthat match the attributes of the advertisement opportunity.
 20. A systemcomprising: one or more processors of a machine; and a hardware storagedevice storing instructions that when executed by the one or moreprocessors, causes the machine to perform operations comprising:accessing media to be communicated to a user device of a viewer;accessing a communicatively coupled database that stores attributes tosearch for in the media, the attributes to search for in the mediacomprising attributes of an object of an advertisement opportunity inwhich an advertisement is to be embedded; performing patternrecognition, using pattern recognition software, to identify the objectin the media that includes the attributes from the database; retrieving,from a second database, valuation information for the advertisementopportunity, the valuation information indicating a value for theadvertisement opportunity based on one or more of the attributes of theadvertisement, including at least a plot line attribute and an activityattribute, wherein the plot line attribute is a function of acriticality of an associated scene in a plot line comprising a pluralityof scenes, wherein the value is higher for critical scenes compared tonon-critical scenes, and wherein the activity attribute is associatedwith a determined level of activity of the scene; determining the valuefor the advertisement opportunity based on the valuation information;offering the advertisement opportunity to a potential advertiser basedon the overall value determined using the attributes of theadvertisement opportunity; in response to the potential advertiseraccepting the offering, graphically embedding an advertisementassociated with the potential advertiser into a display area of theobject based on placement and contour data for the object generated bythe pattern recognition software; and transmitting the media through amedia distribution network to the use device of the viewer.